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ABSTRACT 


Department  of  Defense  (DOD)  spending  has  been  steadily  inereasing  ever  since  the  early 
1990s.  During  that  period,  the  acquisition  workforce  has  steadily  declined.  This  situation 
resulted  in  an  undermanned  and  undertrained  contracting  workforce  with  an  increased 
workload.  With  the  workforce  spread  thin,  lapses  in  contracting  processes  occurred.  As  a 
result  of  these  issues,  in  2008,  the  DOD  established  the  requirement  for  independent 
management  reviews,  or  peer  reviews,  of  contractual  actions. 

Since  the  onset  of  the  peer-review  requirement,  the  Defense  Procurement 
Acquisition  Policy  (DPAP)  has  maintained  a  database  of  peer-review  results.  Data 
analytics  were  used  to  analyze  the  frequency  of  occurrences  of  the  data  elements  within 
the  DPAP  database  of  peer-review  results  in  an  effort  to  answer  two  research  questions. 
First,  are  there  trends  within  the  peer-review  results  of  DOD-level  peer-reviewed 
contracts?  Second,  are  any  trends  identified  related  to  the  competency  gaps  identified  in 
the  2007  Department  of  Defense  Contracting  Worlrforce  Competency  Assessment  Final 
Report! 

Trends  within  the  data  elements  present  in  the  DPAP  database  of  peer-review 
results  were  identified.  Certain  categories  garnered  more  attention  of  the  peer-review 
teams.  Based  on  trends  identified,  recommendations  are  provided  to  improve  the  overall 
usefulness  of  the  DPAP  database  of  peer-review  results. 
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I.  INTRODUCTION 


A,  BACKGROUND 

Department  of  Defense  (DOD)  spending  has  been  steadily  increasing  ever  since 
the  early  1990s.  During  the  same  time  period,  the  acquisition  workforce  has  steadily 
declined.  The  2007  Gansler  Commission  report  on  Army  contracting  noted  that  over  a 
12-year  period,  contracting  actions  had  risen  350%  while  the  Army  contracting 
workforce  had  been  reduced  by  50%  (Gansler,  2007).  This  situation  resulted  in  an 
undermanned  and  undertrained  contracting  workforce,  with  a  vastly  increased  workload. 
Within  the  DOD,  workload  per  GS-1102  (Contract  Specialist)  staff  member  rose  from 
an  annual  average  of  $6.4  million  in  contract  actions  in  fiscal  year  (FY)  1996  to  nearly 
$13  million  in  FY  2005  (Girovasi,  2007). ^  With  the  onset  of  overseas  contingency 
operations  in  Iraq  and  Afghanistan,  demand  for  the  acquisition  workforce  spiked.  The 
increase  in  spending,  along  with  the  corresponding  decrease  in  the  acquisition  workforce, 
is  highlighted  in  Figure  1. 


1  As  the  dollar  value  of  a  contract  action  increases,  so  does  the  workload  required  to  process  it.  For 
example,  micropurchase  procedures  apply  under  $3,000,  and  simplified  acquisition  procedures  apply  from 
$3,000  up  to  $150,000  (FAR  13.003(b)(1),  2014).  Additional  workload  is  generated  when  a  contractor  is 
required  to  submit  Certified  Cost  or  Pricing  Data  for  purchases  above  $700,000  when  no  exceptions  apply 
(FAR  15.403-4(a)(l),  2014). 


1 


Fiscal  Year 

■  DoO  Procurenrvent  Budget  ♦  Acquisition  Organization  Worltforce 

Figure  1.  DOD  Acquisition  Trends  (from  Gansler,  2007) 

With  the  workforce  spread  thin,  lapses  in  contracting  processes  began  to  occur.  A 
Government  Accountability  Office  (GAO)  report,  GAO-05-274,  Contract  Management: 
Opportunities  to  Improve  Surveillance  on  DOD  Service  Contracts,  issued  in  March  2005, 
reviewed  90  service  contracts.  Out  of  these  90  contracts,  the  GAO  (2005)  found 
insufficient  surveillance  on  24.  From  those  24  contracts,  15  had  no  surveillance 
whatsoever  (GAO,  2005).  The  Gansler  Commission  report  revealed  that,  as  of  2007, 
83  Army  criminal  investigations  relating  to  contract  fraud  were  ongoing  in  Iraq, 
Afghanistan,  and  Kuwait.  At  the  same  time,  23  government  employees  (both  civilian  and 
military)  were  charged  or  indicted  in  federal  court.  The  contracts  affected  represented 
over  $6  billion  in  value  (Gansler,  2007). 

As  a  result  of  these  and  other  issues  discussed  during  the  literature  review  chapter, 
the  DOD  established  the  requirement  for  independent  management  reviews,  or  peer 
reviews,  of  contractual  actions  in  2008.  The  purpose  of  the  peer-review  requirement  was 
to  help  address  the  shortfalls  in  manning  and  experience  within  the  DOD  contracting 
community  by  ensuring  policy  and  regulations  were  followed  in  a  consistent  manner,  and 
that  the  available  experience  was  shared  across  the  contracting  workforce  (Assad,  2008). 
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Identifying  and  tracking  the  trends  within  the  data  provided  by  the  peer-review  process  is 
critical  to  identifying  systemic  problems  within  both  the  training  of  the  DOD’s 
contracting  workforce  and  how  the  contracting  workforce  is  operating. 


B.  PURPOSE 

The  primary  purpose  of  this  project  was  to  examine  the  Defense  Procurement 
Acquisition  Policy  (DPAP)  database  of  peer-review  results  and  identify  any  trends  within 
the  data.  This  project  was  not  intended  to  be  an  exhaustive  study  of  the  DPAP  database  of 
peer-review  results.  Rather,  it  was  contemplated  as  a  way  to  identify  overarching  trends 
for  further  analysis  in  the  future.  There  is  one  secondary  goal  associated  with  this  project: 
examine  any  relationships  between  trends  observed  within  the  DPAP  database  of  peer- 
review  results  and  the  competency  gaps  by  the  Contracting  Competency  Model  (CCM)  in 
the  Department  of  Defense  Contracting  Workforce  Competency  Assessment  Final  Report 
(DPAP,  2007). 

C.  RESEARCH  QUESTIONS 

In  an  effort  to  investigate  a  previously  unexplored  topic,  I  crafted  two  research 
questions: 

•  Are  there  trends  within  the  peer-review  results  of  DOD-level  peer- 
reviewed  contracts? 

•  Are  any  trends  identified  related  to  the  competency  gaps  identified  in  the 
Department  of  Defense  Contracting  Workforce  Competency  Assessment 
Final  Report  (DPAP,  2007)? 

D.  BENEFITS  AND  LIMITATIONS  OF  THIS  PROJECT 

The  results  of  this  project  can  be  used  to  target  and  strengthen  areas  of  training 
that  are  leading  to  systemic  issues  within  the  contracting  process.  It  can  also  be  used  to 
identify  areas  of  the  contracting  process  that  are  simply  being  neglected  and  would 
benefit  from  additional  emphasis  by  contracting  officers  and  their  supervisors.  Improved 
understanding  of  the  trends  within  the  DPAP  database  of  peer-review  results  will  enable 
policymakers  to  structure  decisions  appropriately  to  increase  both  the  efficiency  and 
effectiveness  of  the  DOD  contracting  workforce. 
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The  limitation  of  this  project  rests  with  the  fact  there  were  only  288  entries  within 
the  DPAP  database  of  peer-review  results.  Thus,  it  was  not  possible  to  obtain  a  larger 
population  or  sample  size  to  analyze. 

E.  RESEARCH  METHODOLOGY 

The  research  began  with  a  literature  review.  The  literature  review  addresses  why 
DOD  conducts  peer  reviews  and  addresses  corporate  procurement  review  practices, 
development  of  the  peer-review  requirement,  and  guidance  on  conducting  peer  reviews. 
I  approached  the  research  questions  by  analyzing  the  DPAP  database  of  peer-review 
results  in  terms  of  frequency  of  occurrences  of  various  data  aspects,  including  Category, 
Type  of  Contract,  Review  Phase,  and  Type  of  Feedback.  I  examined  the  Narrative 
comments  section  from  a  qualitative  standpoint  to  identify  any  recurring  themes  within 
the  comments. 

F.  ORGANIZATION  OF  REPORT 

The  report  consists  of  five  chapters.  Chapter  I  contains  a  brief  background,  the 
purpose  of  the  research,  research  questions  examined,  relative  benefits  and  limitations  of 
the  project,  and  an  overview  of  the  research  methodology.  Chapter  II  is  the  literature 
review,  which  is  primarily  concerned  with  addressing  the  questions  of  why  we  peer 
review  contractual  actions,  how  the  requirement  was  developed,  and  how  it  has  been 
implemented.  To  that  end,  the  history  of  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  is  examined.  I  review  multiple  GAO  and  commission 
reports  that  reveal  the  decline  in  the  experience  and  manning  of  the  acquisition 
workforce,  coupled  with  the  rise  in  spending  activities  and  the  resultant  issues  that 
caused.  Chapter  III  discusses  where  the  data  originated  and  how  it  was  analyzed.  Chapter 
IV  provides  the  results  from  the  analysis  of  the  data.  Chapter  V  contains  the  summary  of 
the  project,  recommendations  for  improving  the  DPAP  database  of  peer-review  results, 
and  areas  for  further  research  relating  to  the  findings  of  this  project. 
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G.  SUMMARY 

Chapter  I  provided  the  baekground  information  on  the  peer-review  requirement 
and  the  purpose  of  the  project,  stated  the  research  questions  and  the  benefits  and 
limitations  of  the  project,  and  briefly  addressed  the  methodology  and  organization  of  the 
written  report.  The  next  chapter  contains  the  literature  review.  The  literature  review 
addresses  why  the  DOD  conducts  peer  reviews  of  contractual  actions.  It  also  addresses 
corporate  procurement  review  practices,  development  of  the  peer-review  requirement, 
and  guidance  on  conducting  peer  reviews. 
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II.  LITERATURE  REVIEW 


A,  INTRODUCTION 

The  previous  chapter  discussed  the  basis  of  this  research.  In  this  chapter,  I 
examine  why  the  DOD  conducts  peer  reviews  of  contractual  actions.  I  also  examine 
development  of  the  peer-review  requirement,  and  guidance  on  conducting  peer  reviews. 
Elements  of  this  chapter  include  explanations  of  the  DOD’s  initiation  of  required  peer 
reviews,  the  definition  of  a  peer  review,  and  the  execution  of  peer  reviews.  Additionally, 
I  review  how  both  public  and  private  procurement  agencies  evaluate  performance. 

B,  WHY  PEER  REVIEW? 

1.  Introduction 

To  understand  the  reasoning  behind  the  peer-review  requirement,  it  is  necessary 
to  look  back  to  the  beginnings  of  the  modem  defense  acquisition  workforce,  which  can 
be  traced  back  to  the  early  1990s,  with  the  passage  of  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA,  1990).  The  history  since  1990  develops  a  story  of  the 
development  of  a  tmly  professional  workforce  and  how  it  ran  into  issues  of  cutbacks  in 
an  era  of  persistent  defense  spending  cuts.  This  led  to  a  situation  of  neglect  within  the 
acquisition  workforce  in  a  time  of  persistent  conflict  when  the  workforce  could  have  been 
utilized  to  its  fullest,  ultimately  causing  the  workforce  to  break  down  under  the  strain  of 
too  many  requirements  and  too  few  trained  personnel. 

2,  Defense  Acquisition  Workforce  Improvement  Act 

As  a  result  of  an  acquisition  workforce  that  was  plagued  by  scandal  and 
inefficiencies,  the  DAWIA  was  developed  and  enacted  into  law  in  1990.  Fraud,  waste, 
and  abuse  have  gone  hand  in  hand  with  military  operations  throughout  history.  The  U.S. 
government’s  response  after  each  conflict  has  been  to  enact  progressively  more 
restrictive  legislation  in  the  name  of  acquisition  reform.  Defense  historian  William 
Gregory  (1989)  described  in  his  work  The  Defense  Procurement  Mess  the  state  of 
acquisitions  in  the  late  1980s  “as  one  that  had  been  managed  and  over-reformed  into 
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impotence  with  volumes  of  oversight  regulations.”  Scandals  dragged  the  issues  further 
into  the  spotlight.  Operation  Ill  Wind,  a  Department  of  Justice  probe  that  uncovered 
widespread  corruption  and  incidents  of  fraud  and  bribery  within  the  defense  procurement 
system,  was  concluded  in  1988.  In  all,  the  operation  resulted  in  more  than  60  convictions, 
including  that  of  former  Assistant  Secretary  of  the  Navy  Melvin  Paisley  (Layton,  2007). 

Voted  into  law  as  Public  Law  101-510  on  November  5,  1990,  the  DAWIA  was 
codified  in  Title  10,  Chapter  87  of  the  U.S.  Code  and  amended  in  1991,  1993,  2001, 
2003,  2008,  2011,  and  2013  (DAWIA,  2013).  The  purpose  of  the  DAWIA  was  to  shift 
the  focus  from  regulating  acquisition  procedures  to  developing  a  professional  acquisition 
workforce.  It  established  a  certification  process  for  contracting  as  a  career  field,  to 
include  establishing  standards  for  education,  training,  and  progressive  levels  of 
experience.  DAWIA  also  prescribed  the  creation  of  a  center  for  defense  acquisition 
education.  The  Defense  Acquisition  University  (DAU)  was  formed  in  1992  to  satisfy  that 
requirement.  The  DAU  was  initially  established  from  a  consortium  of  existing  DOD 
organizations,  bound  together  by  a  memorandum  of  agreement  to  cooperate  as  the  DAU. 
The  DAU  has  been  highly  successful  in  raising  professional  standards  of  education 
for  acquisition  professionals;  however,  while  quality  has  increased,  quantity  has  not 
(Layton,  2007). 

3.  The  Acquisition  Workforce  in  Decline 

Much  has  been  written  over  the  past  six  years  about  issues  facing  the  acquisition 
workforce.  The  Acquisition  Advisory  Panel  (AAP)  issued  a  report  in  2006  through  the 
Office  of  Federal  Procurement  Policy  (OFPP)  detailing  the  challenges  facing  the 
acquisition  workforce.  The  acquisition  career  field  was  not  spared  from  the  larger  defense 
drawdown  of  the  early  to  mid-1990s.  Actual  numbers  for  the  time  period  are  difficult  to 
find,  for  two  main  reasons.  First,  each  organization  in  the  federal  government  defines 
“acquisition  professional”  differently;  and  second,  prior  to  1999,  there  was  no 
requirement  mandating  tracking  acquisition  professionals  separately  within  the  federal 
employment  system.  What  is  known  is  that  the  acquisition  workforce  went  through 
substantial  reductions  throughout  the  1990s,  and  little  to  no  hiring  took  place. 
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Meanwhile,  the  workforce  continued  to  gain  experience  while  it  edged  ever  closer  to 
retirement  (OFPP,  2007).  Figure  2  illustrates  the  issue,  with  a  large  number  of  contracting 
officers  with  less  than  10  years  of  experience,  with  a  dip  in  the  mid-level  experience 
range,  followed  by  a  large  number  of  employees  at  the  retirement  age. 


Defense  Acquisition  Workforce  -  DOD-Wide 

Distribution  by  Years  to  Retirement  Eligibility  (Civiiians)(FY09) 
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Figure  2.  Defense  Acquisition  Workforce  Retirement  Eligibility  Distribution 

(from  DOD,  2010,  p.2-22) 


Since  1999,  the  differences  across  the  federal  government  in  defining  acquisition 
professionals  make  it  difficult  to  ascertain  precise  statistics  about  growth  trends  within 
the  workforce,  though  all  numbers  point  toward  decline,  ranging  anywhere  from  3%  to 
27%  (OFPP,  2007).  Within  the  current  acquisition  workforce,  50%  were  retirement 
eligible  in  2010  (Girovasi,  2007).  Another  well-documented  fact  is  the  increase  in 
acquisition-related  spending  since  1990.  Paradoxically,  as  the  acquisition  workforce  was 
reduced,  the  reliance  on  the  workforce  increased.  Consider  some  examples: 

•  Federal  acquisition  expenditures  tripled  from  FY  1991  to  FY  2006, 
reaching  a  level  of  $424  billion. 
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•  Federal  aequisition  spending  inereased  by  65%  from  FY  2001  to  FY  2005, 
representing  an  inerease  from  $235  billion  to  $388  billion. 

•  Within  the  DOD,  workload  per  GS-1102  (Contract  Specialist)  staff 
member  rose  from  an  annual  average  of  $6.4  million  in  contract  actions  in 
FY  1996  to  nearly  $13  million  in  FY  2005. 

•  Contracts  for  services  accounted  for  60%  of  total  spending  in  FY  2005  and 
2006  (Girovasi,  2007). 

The  issue  at  hand  becomes  one  of  more  contract  actions  to  process  with  fewer, 
less-experienced  contracting  officers  available  to  process  them.  The  2007  Gansler 
Commission  report  on  Army  contracting  noted  that  over  a  12-year  period,  contracting 
actions  had  risen  350%  while  the  Army  contracting  workforce  had  been  reduced  by  50% 
(Gansler,  2007). 

4,  The  System  Breakdown 

Several  reports,  documents,  and  commissioned  studies  have  established  what 
happened  once  the  contracting  system  was  placed  under  stress.  GAO-02-737  was  issued 
in  July  2002,  approximately  10  years  after  the  overhaul  of  the  acquisition  system  began, 
but  before  the  United  States’  involvement  in  Iraq  and  Afghanistan  and  the  surge  in 
contracting  actions.  The  GAO  (2002)  noted  that  when  the  DOD  had  adopted  a  definition 
of  the  acquisition  workforce,  it  was  very  multidisciplinary  in  nature,  covering  various 
functions  of  the  contracting  team.  Various  government  civilian  agencies,  by  contrast,  had 
not  taken  such  a  broad  approach  and  only  considered  the  GS-1 102  series  to  make  up  the 
contracting  team.  As  a  result  of  the  DOD  outlook,  they  had  developed  broad  training  and 
tracking  programs  to  assist  in  maintaining  and  accounting  for  their  acquisition 
professionals.  The  GAO  (2002)  found  overall  that  the  DOD  was  executing  the  DAWIA 
reforms  well;  however,  the  government  civilian  agencies  still  had  work  to  do.  The  report 
also  found,  through  the  interview  process,  that  acquisition  leadership  felt  that  funding 
was  currently  adequate  to  maintain  training  proficiency  levels,  but  that  leadership  worried 
about  upcoming  budget  cuts,  particularly  to  the  DAU’s  budget  (GAO,  2002). 

By  2005,  the  situation  seemed  to  have  worsened.  GAO-05-274,  Contract 
Management:  Opportunities  to  Improve  Surveillance  on  DOD  Service  Contracts,  issued 

in  March  2005,  reviewed  90  service  contracts.  Out  of  these  90  contracts,  the  GAO  found 
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insufficient  surveillance  on  24.  From  those  24  contracts,  15  of  them  had  no  surveillance 
whatsoever.  The  DOD  readily  acknowledged  the  lapse,  and  in  response  to  the  findings 
stated  they  simply  did  not  have  enough  qualified  contracting  personnel  to  fulfil  the 
requirement  (GAO,  2005). 

Testimony  by  David  M.  Walker,  Comptroller  General  of  the  U.S.,  before 
the  House  of  Representatives  Subcommittee  on  Defense  and  the  Committee  on 
Appropriations  in  September  2006,  was  captured  in  GAO-06-800T  (Walker,  2006).  His 
testimony  was  based  on  six  years  of  research  using  the  generally  accepted  government 
auditing  standards  (GAGAS).  The  work  revealed  that  dollar  values  for  major  weapon 
system  and  service  contracts  were  indeed  on  a  sharp  rise,  and  the  workforce  had 
remained  flat.  Issues  of  insufficient  oversight  of  contractors,  weak  business  practices,  and 
poor  incentives  for  contractors  to  perform  well  were  also  identified  (Walker,  2006). 
Figure  3  illustrates  the  issue;  the  line  within  the  figure  demonstrates  the  acquisition 
workforce,  while  the  columns  represent  the  increase  in  obligations. 
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Figure  3.  DOD  Contract  Obligations  and  Acquisition  Workforce  Trends 

(from  GAO,  2006,  p.  4) 
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In  late  2006,  the  GAO  also  eonvened  a  forum  of  experts  on  aequisition  and 
management  from  government,  academia,  and  the  public  sector  to  discuss  overall  federal 
acquisition  challenges  over  the  longer  term.  GAO-07-45SP  found  that  leaders  among  the 
various  federal  agencies  were  not  recognizing  the  important  role  the  acquisition 
workforce  played  in  their  organizations.  It  also  upheld  DOD  contract  management  as  a 
federal  government  high-risk  area,  a  status  it  had  held  since  1992  (GAO,  2007). 

As  stories  of  fraud,  waste,  and  abuse,  coupled  with  rampant  spending  in  Iraq, 
Afghanistan,  and  Kuwait,  continued  to  mount,  the  Secretary  of  the  Army  established  a 
commission  to  examine  expeditionary  contracting  in  2007.  The  commission  was  headed 
by  Dr.  Jacques  S.  Gansler,  former  Undersecretary  of  Defense  for  Acquisitions, 
Technology  &  Logistics.  The  Gansler  Commission  found,  as  several  previous  GAO 
reports  show,  that  the  contracting  workforce  had  become  short-staffed  and  undertrained. 
It  also  highlighted  massive  amounts  of  fraud  relating  to  government  contracts,  mostly  in 
Iraq.  As  of  October  23,  2007,  83  Army  criminal  investigations  relating  to  contract  fraud 
were  ongoing  in  Iraq,  Afghanistan,  and  Kuwait.  At  the  same  time,  23  government 
employees  (both  civilian  and  military)  were  charged  or  indicted  in  federal  court.  The 
contracts  affected  represented  over  $6  billion  in  value  (Gansler,  2007). 

Further  testimony  before  Congress  by  David  M.  Walker  was  captured  in  two 
additional  GAO  reports,  GAO-07-1098T  and  GAO-08-621T  (GAO  2007;  2008).  These 
reports  continued  to  highlight  issues  of  shortages  in  the  workforce  and  insufficiently 
trained  acquisition  personnel.  Additionally,  one  report  noted  the  dramatic  increase  in  the 
role  of  service  contracts  within  the  DOD  and  a  shift  away  from  the  utilization  of 
government  employees.  Concerns  were  also  raised  about  the  line  becoming  blurred 
between  contractor  involvement  with  inherently  governmental  functions.  The  marked 
increase  in  the  number  of  service  contracts  only  multiplied  the  stress  on  the  government 
contracting  workforce  (GAO,  2008).  A  report  delivered  to  Congress  in  2007  by  John  P. 
Hutton,  Director  of  Acquisition  and  Sourcing  Management,  reinforced  this  by  revealing 
that  the  DOD  had  awarded  service  contracts  for  security  guard  services  at  57  domestic 
bases,  and  that  46  of  them  were  awarded  on  a  sole-source  basis.  The  cost  of  the  contracts 
rose  by  25%  compared  to  when  the  same  contracts  were  previously  awarded 
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competitively.  The  same  report  identified  that,  indexed  to  2006  dollars,  the  amount 
awarded  for  DOD  serviee  eontraets  rose  from  $85.1  billion  in  1996  to  $151  billion  in 
2006,  representing  a  78%  inerease  (Hutton,  2007). 

Of  speeifie  mention,  the  GAO  issued  the  report  titled  Status  of  DOD’s 
Implementation  of  Independent  Management  Reviews  for  Service  Acquisitions  in  January 
2010.  Overall,  the  report  found  that  there  are  still  issues  with  the  implementation,  but  the 
DOD  is  making  progress.  The  report  noted  two  major  issues.  First,  at  the  time  of  the 
report,  the  military  departments  stated  that  they  had  undertaken  hundreds  of  reviews; 
however,  they  eould  not  deliver  preeise  numbers  beeause  of  a  laek  of  a  defined  reporting 
proeess.  Seeond,  and  perhaps  equally  grievous,  the  DOD  had  not  yet  developed  a 
methodology  or  meehanism  to  report  review  results  and  lessons  learned  baek  to  the  foree 
(GAO,  2010).  The  following  seetion  addresses  the  development  of  the  peer  review- 
requirement. 

C.  DEVELOPMENT  OF  PEER-REVIEW  REQUIREMENT 

The  development  of  the  peer-review  requirement  began  in  2008  with  a  DOD 
memorandum  informing  the  serviee  branehes  that  eontraetual  aetions  over  a  eertain  dollar 
threshold  would  be  subjeet  to  a  DOD-level  peer-review  proeess,  both  pre-  and  post- award 
of  the  eontraet.  It  also  instrueted  the  respeetive  serviees  to  establish  internal  peer-review 
proeesses  based  on  lower  dollar  thresholds.  The  DFARS  and  Army  Federal  Aequisition 
Regulation  Supplement  (AFARS)  (2014)  provided  speeifie  guidanee  about  the 
implementation  of  the  new  poliey,  and  eontraeting-speeifie  guides  sueh  as  the  Contract 
Attorney’s  Desk  Guide  and  the  Army  Contracting  Command  Desk  Book  provided 
referenees  to  the  new  regulations. 

1.  DOD  Peer-Review  Implementation  Memorandum 

The  DPAP  organization  is  the  poliey  arm  of  DOD  for  defense  proeurement  and 

aequisitions.  This  organization  is  responsible  for  “all  Contraeting  and  Proeurement  poliey 

matters  ineluding  e-Business  in  the  DOD.  DPAP  exeeutes  that  poliey  through  the  timely 

update  of  the  DFARS,  PGI,  and  5000. 1&2”  (DPAP,  2014).  The  DOD  established  the 

requirement  for  eontraet  peer  reviews  by  issuing  a  poliey  memorandum  titled  “Peer 
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Reviews  of  Contracts  for  Supplies  and  Services”  dated  September  29,  2008  (Assad, 
2008).  The  memorandum  stated  three  primary  objectives:  (1)  to  ensure  that  Contracting 
Officers  across  the  Department  are  implementing  policy  and  regulations  in  a  consistent 
and  appropriate  manner;  (2)  to  continue  to  improve  the  quality  of  contracting  processes 
across  the  Department;  3)  to  facilitate  cross-sharing  of  best  practices  and  lessons  learned 
across  the  Department  (Assad,  2008). 

The  peer-review  requirement  grew  from  two  sources.  First,  the  DOD  had 
implemented  a  pilot  program  to  evaluate  proposed  contract  awards,  focusing  on  the  pre¬ 
contract  award  process.  Second,  Congress  enacted  Section  808  of  the  National  Defense 
Authorization  Act  (NDAA)  for  Fiscal  Year  2008,  Public  Law  110-181.  This  law  required 
the  DOD  to  establish  requirements  for  post-award  independent  management  reviews  of 
contracts  for  services,  and  for  sharing  lessons  learned  from  those  reviews  (NDAA,  2008). 
The  existing  peer-review  pilot  program  expanded  to  satisfy  the  requirement  brought 
about  by  the  NDAA  of  2008. 

Rather  than  being  regulatory  in  nature,  the  peer  reviews  were  envisioned  as  a 
quality  control  and  an  advisory  tool  for  contracting  officers.  It  is  important  to  note  that 
the  agency  managing  the  procurement  still  has  the  final  decision  on  how  the  procurement 
is  executed,  though  the  memorandum  did  establish  a  requirement  that  all  peer-review 
recommendations  and  their  dispositions  be  documented  in  the  contract  file. 

The  DPAP  office  has  responsibility  for  organizing  review  teams  and  facilitating 
pre-award  peer  reviews  for  all  contracts  valued  at  or  above  $1  billion,  and  post-award 
reviews  of  service  contracts  valued  at  or  above  $  1  billion.  The  contract  value  should  also 
include  the  estimated  value  of  any  options  associated  with  the  contract.  The  DPAP-level 
peer-review  teams  consist  of  senior-level  DOD  contracting  officials,  both  civilian  and 
military,  and  members  of  the  Office  of  General  Counsel. 

Further,  the  initial  memorandum  directed  that  each  military  department,  defense 
agency,  and  DOD  field  activity  would  publish  its  own  policies  for  the  conduct  of  pre-  and 
post-award  peer  reviews  for  contracts  valued  at  less  than  $1  billion. 
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2,  Defense  Federal  Acquisition  Regulation  Supplement  Peer-Review 
Requirements 

The  DOD  codified  the  requirement  into  the  Defense  supplement  to  the  Federal 
Acquisition  Regulation  (FAR).  The  FAR  is  a  detailed  federal  regulation  that  prescribes 
how  the  federal  government  will  procure  supplies,  services,  and  equipment.  The  DOD 
has  its  own  supplement  to  the  FAR,  as  does  each  service  component  (Army,  Navy,  and 
Air  Force).  Defense  Federal  Acquisition  Regulation  (DFAR)  Part  201.170  explains  the 
requirement  for  peer  reviews: 

201,170  Peer  Reviews, 

(a)  DOD  peer  reviews. 

(1)  The  Office  of  the  Director,  Defense  Procurement  and  Acquisition 
Policy,  will  organize  teams  of  reviewers  and  facilitate  peer  reviews  for 
solicitations  and  contracts,  as  follows  using  the  procedures  at  PGI 
201.170— 


(i)  Pre-award  peer  reviews  for  competitive  procurements  will  be 
conducted  in  three  phases  for  all  solicitations  valued  at  $1  billion 
or  more; 

(ii)  Pre-award  peer  reviews  for  noncompetitive  procurements  will 
be  conducted  in  two  phases  for  new  contract  actions  valued  at 
$500  million  or  more;  and 

(iii)  Post-award  peer  reviews  will  be  conducted  for  all  contracts  for 
services  valued  at  $1  billion  or  more. 

(2)  To  facilitate  planning  for  peer  reviews,  the  military  departments  and 
defense  agencies  shall  provide  a  rolling  annual  forecast  of  acquisitions  that 
will  be  subject  to  DOD  peer  reviews  at  the  end  of  each  quarter  (i.e.,  March 
31;  June  30;  September  30;  December  31),  to  the  Deputy  Director, 
Defense  Procurement  and  Acquisition  Policy  (Contract  Policy  and 
International  Contracting)  via  e-mail  to  osd,pentagon,ousd-atl,mbx,peer- 
reviews@mail. 


15 


(b)  Component  peer  reviews.  The  military  departments  and  defense  ageneies  shall 
establish  proeedures  for — 

(1)  Pre-award  peer  reviews  of  solieitations  for  eompetitive  proeurements 
valued  at  less  than  $1  billion; 

(2)  Pre-award  peer  reviews  for  noneompetitive  proeurements  valued  at 
less  than  $500  million;  and 

(3)  Post-award  peer  reviews  of  all  eontracts  for  services  valued  at  less  than 
$1  billion. 

The  DFARS  mentions  peer  reviews  in  Subpart  215.270,  Solicitation  and  Receipt 
of  Proposals  and  Information;  Subpart  207.104,  General  Procedures  of  Acquisition 
Planning;  and  Subpart  237.102,  Policy  of  Service  Contracts,  but  only  to  the  extent  of 
referring  back  to  Subpart  201.170.  The  FAR,  in  and  of  itself,  makes  no  reference  to  peer 
reviews.  Since  peer  reviews  are  a  DOD  requirement,  there  is  no  regulatory  requirement 
for  the  policy  on  peer  reviews  to  appear  in  the  FAR. 

3.  Overview  of  Contract  Attorney’s  Desk  Book  Peer-Review 
Requirements 

The  2013  Contract  Attorney’s  Desk  Book  (CADB),  published  by  the  Contract  and 
Fiscal  Law  Department  of  the  DOD’s  Judge  Advocate  General’s  School,  highlights  the 
need  for  peer  reviews.  Chapter  8,  Negotiated  Procurements  and  Source  Selection, 
mentions  fairly  early  that  peer  reviews  are  a  planning  consideration  to  be  considered  in 
the  acquisition  planning  phase,  and  reiterates  the  requirements  stated  in  DFARS  201.170. 
The  CADB  does  not  provide  any  more  details  than  what  is  included  in  the  DFARS — only 
that  it  is  a  statutory  requirement  {CADB,  2013).  The  next  section  examines  the 
procedures,  guidance,  and  information  (PGI)  relating  to  the  peer-review  requirement. 

D,  PROCEDURES,  GUIDANCE,  AND  INFORMATION  201.170 
CONDUCTING  PEER  REVIEWS 

The  DFARS  (2014)  contains  implementation  and  DOD-specific  supplementation 
information  to  the  FAR  (2014).  The  DFARS  sets  forth  requirements  established  by  law, 
policies  that  are  implemented  across  the  DOD,  delegation  authority,  DOD-specific  FAR 
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deviations,  and  other  policies  deemed  to  have  a  significant  impact  on  the  public.  The  PGI 
series  offers  supplementation  to  the  DFARS.  The  PGI  contains  techniques,  procedures, 
and  guidance  of  a  non-regulatory  nature  that  are  not  included  in  the  DFARS  (DPAP, 
2013b). 

PGI  20 1. 1 70,  Conducting  Peer  Reviews,  provides  guidance  on  how  to  conduct 
peer  reviews  in  accordance  with  DFARS  201. 170.  Since  the  PGI  201. 170  is  issued  as 
guidance,  rather  than  strict  regulation,  it  provides  a  view  into  what  the  DPAP  considers 
most  important  within  the  peer-review  process.  Consider  the  longest  paragraph  in  the 
section: 


The  results  and  recommendations  that  are  products  of  peer  reviews  are 
intended  to  be  advisory  in  nature;  however,  in  the  event  the  peer  review 
report  includes  a  recommendation  that  is  identified  as  “significant”  and  the 
contracting  officer  does  not  intend  to  follow  that  recommendation,  the 
senior  procurement  official  of  the  contracting  activity  for  the  reviewed 
organization  must  be  made  aware  of  this  fact  before  action  is  taken  (or 
inaction,  as  applicable)  that  is  contrary  to  the  recommendation.  Reviews 
will  be  conducted  in  a  manner  that  preserves  the  authority,  judgment,  and 
discretion  of  the  contracting  officer  and  the  senior  officials  of  the 
acquiring  activity.  (DPAP,  2013b) 

This  section  would  seem  to  indicate  the  importance  placed  on  the  peer-review 
results.  It  also  highlights  an  interesting  dichotomy  between  the  sanctity  of  the  contracting 
officer  and  the  recommendations  of  the  peer-review  board.  It  is  explicit  in  this  statement 
that  the  contracting  officer  can  stand  by  his  or  her  decision,  though  it  will  be  under  the 
spotlight. 

PGI  20 1. 1 70  begins  by  stating  the  review  criteria  that  are  the  tenets  of  DOD-wide 
contractual  actions,  and  includes  a  document  highlighting  these  areas.  This  is  the  core  of 
the  peer-review  process  and  the  benchmark  against  which  peer  reviews  are  conducted. 
Figures  4  and  5  describe  what  participants  in  the  peer-review  process  are  examining  for 
pre-  and  post-award  acquisition  of  services. 
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Figure  4.  DOD  Peer-Review  Criteria  for  Services  Acquisitions  during  the 

Pre-award  Phase  (from  DPAP,  2013  c) 


Dol)  Ciitenn  for  the  Acquisition  of  SeiAices  ( 

1 

Tenet 

Review  Criteria: 

Primarily 

Assessed  at: 

1 

Acquisition  Strategy:  Service  acquisitions  should  have  a  comprehensive  acquisition  strategy  that 
reflects  program  ob)ecttves.  leverages  spend  data  to  arrive  at  strategic  sourcing  solutions  fa  the 
enterprise  being  supported,  incorporates  strategic  contracting  tools,  is  developed  pria  to  the 
issuance  of  a  sdiotation  (amended  as  applicable)  and  is  actiered  to  throu^out  performance 

Detailed  written  and 
approved  acquisibon 
strategy 

Review/Approval  of 
Acquisibon  Strategy 

2 

Clearly  Defined  Requirements:  Service  acquisitions  should  use  a  performance  woric  statement  or 

statement  of  objectives  that  clearly  defines  the  services  the  pro^m  seeks  to  receive 

Requirements  clearly 
stated 

Peer  Review 

3 

Period  of  Performance:  Ckintract  penod  of  performance  should  reflect  a  length  that  is  consistent 

with  technologcal  dependence,  industry  standards,  and  suffdent  time  to  reclaim  program  ownership 
On  cases  with  an  acquisiton  history  of  a  single  provider)  such  that  fair  competition  can  occur 

Typically  this  is  3-5  years  with  certain  exceptions  (e  g  performance-based  logistics  and  energy- 
savings  performance  contracts)  Single-award  contracts  fa  knowledge  based  services  should  be 
limited  to  three  years  Onclucing  options)  unless,  by  exception,  fully  justified  fa  longer  periods 

Length  of  contract 
appropriate 

Review/Approval  of 
Acquisibon  Strategy 

4 

Appropriate  Contract  T ype:  Services  acquisitions  should  be  predsposed  toward  CPFF  a  CPIF 

arrangements  when  robust  competitnn  or  recent  competitive  pnang  history  does  na  exist  to  build 
suffiaent  cost  knowledge  of  those  services  within  that  market  segment  When  robust  competition 
exists,  a  there  is  recent  compietitive  pricing  history.  FFP  should  typically  be  used  Award  fee 
contracts  should  be  limited  as  they  provide  limited  mobvation  fa  cost  dsdplre  T&Misthe  least 
preferable  contract  type  and  must  be  justified  when  used  and  limited  (e  g.  <  10%  of  contract  value) 

Contract  type 
appropriate 

Review/Approval  of 
Acquisibon  Strategy 

5 

Socio-Economic  Considerations:  The  Department  higfily  values  small  business  contnbutions  and 

expects  maxirrum  opportunities  fa  small  business  partidpation 

Small  business 

opportunibes 

Review/Approval  of 

Acquisibon  Strategy 

6 

Participation  Decision  Points:  Acquisitions  with  longer  penods  of  performance,  parbcularty 

multiple  award  contracts,  should  indude  deasion  points  (on  and  offramps)  to  ensure  a  qualified  pool 
of  contractas  that  will  provide  continuous  service  throughout  the  life  of  the  contract 

Decision  points  (on  and 
off-ramps)  fa  longer 
term  contracts 

Review/Approval  of 
Acquisibon  Strategy 

7 

Competition:  Service  acquisibon  requirements  should  be  articulated  in  such  a  way  to  provide  for 

maximum  competibon  and.  fa  multiple  award  contracts,  throughout  the  life  of  the  contract  with 
meaningful  competibon  fa  orders  Evaluabon  factas  are  tied  to  kev  program  requirements 

Approach  provides  fa 
robust  compebbon 

Review/Approval  of 
Acquisibon  Strategy 

8 

Objective  Incentives:  Objedive  cntena  will  be  utilized,  whenever  possible,  to  measure  contract 

peiToimance  Service  contracts  valued  at  $1  bilion  or  mae  shall  indude  provisions  to  achieve 
productivity  imaovements  and  cost  efficiendes  throuchout  the  contract  period 

Award/Incentive  Fee 

set  up  to  reward 
effedive  outcomes 

Review/Approval  of 
Acquisibon  Strategy 

9 

inherently  Governmental  Functions  Accjuisitions  (a  services  must  address  the  extent  of  the 

agency's  reliance  on  contractas  to  perform  fundions  closely  associated  with  inherently 
governmental  fundions  as  recjuired  by  FAR  Subparl  7  5.  and  DFARS  Subpart  207  5 

Required 
determinations 
appropnately  executed 

Peer  Review 

10 

Conflict  of  Interests:  When  one  contracta  will  provide  oversi^t  fa  anaher  contrada  a  dired 

work  to  another  contracta.  the  acquisition  documentabon  should  adi±ess  measures  to 
reduce/eliminate  pdenbal  conflid  of  interest 

Evaluate  finanaal 

interest  of  support 
contractas.  as  required 

Peer  Review 

11 

Performance  Management:  Service  accjuisitions  should  utiize  performance  based  charadenstics 

to  the  maximum  extent  pracbcable  to  include  measures  that  are  diredly  tiedto  program  objedives 
These  measures  should  include  consideration  of  program  cost 

Meaningful 

performance  measures 

Peer  Review 

12 

Contract  Surveillance:  Service  acquisitxxis  must  have  assigned  conbacbng  officer  representabves 

(CORs)  that  use  tailored  quality  assurance  surveillance  plans  to  monita  contracta  performance 

Appropnate  program 
oversight 

Peer  Review 

Figure  5.  DOD  Peer-Review  Criteria  for  Services  Acquisitions  during  the 

Post-award  Phase  (from  DPAP,  2013c) 


DoD  C’liteiin  for  the  Acquisition  of  Sei  \ices  (Postnwjw  d) 


Tenet 

Review  Criteria: 

Primarily 

Assessed  at: 

1 

Contractor  Performance  Assessment:  Service  acquisitions  undergo  periodic  reviews  to  ensure 
the  program  is  on  course  with  respect  to  cost,  schedule  and  performance  requirements;  and  any 
necessary  adjustments  are  made 

Program  meets  or 
exceeds  established 
cost,  schedule  and 
performance  criteria 

Peer  Review 

2 

Maintaining  Competition:  Multiple  award  service  acquisitions  use  contracting  mechanisms, 
including  the  use  of  competition,  the  contract  structure  and  type,  the  definition  of  contract 
requirements,  cost  a  pricing  methods,  the  award  and  negotiation  of  task  orders,  and  management 
and  oversight  mechanisms.  In  competitive  procurements  (to  include  multiple  award  IDIQ  competition 
for  task  orders),  offerors  must  be  given  a  reasonable  amount  of  time  to  submit  a  proposal  (typically 

30  days  or  more).  If  sufficient  time  is  provided  for  offerors  to  respond  and  yet  only  one  offer  is 
received  in  a  competitive  procurement,  contracting  officers  should  not  depend  on  the  standard  at 

FAR  15.403-1(c)(ii)  in  determining  the  price  to  be  fair  and  reasonable. 

Robust  competition  fa 
orders,  appropriate 
contract  type,  well- 
defined  requirements, 
appropriate  cost/pridng 
methods 

Peer  Review 

3 

Contractor’s  Subcontract  Management:  Service  acquisitions  undergo  periodic  reviews  to  monitor 
the  contractor's  use.  management,  and  oversight  of  subcontractors 

Contractor’s 
subcontract 
management  evaluated 

Peer  Review 

4 

Contract  Surveillance:  Service  acquisitions  undergo  periodic  reviews  to  ensure  the  appropnate 
staffinq  of  Government  contract  management  and  oversight  functions  to  include  CORs 

Appropriate 

Government  oversi^t 
personnel  in  place 

Peer  Review 

5 

Assessment  of  Excessive  Pass-Through  Charges:  Service  acquisitions  undergo  periodic  review 

to  evaluate  the  extent  of  any  pass-throu^s,  and  excessive  pass-through  charges  (as  defined  in 
DFARS  252.215-7003/4),  by  the  contractor. 

Contractor's  pass¬ 

through  charges 
evaluated 

Peer  Review 

6 

Inherently  Governmental  Functions:  For  service  acquisitions  under  which  one  contractor  provides 

oversight  fa  services  performed  by  other  contractas,  periodic  reviews  evaluate  the  extent  of  the 
agency's  reliance  on  the  contracta  to  perfam  acquisition  functions  closely  associated  with  inherently 
governmental  functions  as  defined  in  section  2383(b)(3)  of  title  10.  United  States  Code 

Evaluate  appropriate 
performance  of 
contracta  functions  (as 
required) 

Peer  Review 

7 

Conflicts  of  Interest  (Financial):  For  service  acquisitions  under  which  one  contracta  provides 

oversight  fa  services  perfamed  by  other  contractas.  periodic  reviews  evaluate  the  financial  interest 
of  any  prime  contracta  perfaming  acquisition  functions  described  in  paragraph  six  in  any  contract  a 
subcontract  with  regard  to  which  the  contracta  provided  advice  or  recorrmendations  to  the  agency. 

Evaluate  finandal 
interest  of  support 
contractas  (as 
required) 

Peer  Review 

8 

Objective  Incentives:  For  contracts  with  award  and/a  incentive  fee.  objective  criteria  will  be 
utilized,  whenever  possible,  to  measure  contract  performance  Where  objective  criteria  exist,  and  it 
is  appropriate  to  also  incentivize  subjective  elements  of  perfamance,  the  most  appropriate  contract 
type  would  be  a  multiple  incentive  type  contract  containing  both  incentive  and  award  fee  criteria 

Award/incentive  fees 
are  paid  in  a  manner 
consistent  with  DoD 
policy,  a  with  a 
corrective  action  plan  in 
place  to  address  past 
inconsistencies 

Peer  Review 

The  objectives  of  the  peer-review  process  are  simple:  to  ensure  that  DOD 
contracting  officers  are  implementing  policy  and  regulations  appropriately,  to  improve 
contracting  as  a  process  throughout  the  DOD,  and  to  facilitate  the  sharing  of  ideas 
and  best  practices  throughout  the  DOD.  Additionally,  the  DPAP  maintains  a 
database  of  peer-review  results  for  contracts  reviewed  at  the  DOD  level,  located  at 
www.acq.osd.mil/dpap/cpic/cp/peer  reviews.html,  which  is  the  subject  of  further 
analysis  in  this  thesis  (DPAP,  2013a). 

According  to  PGI  201.170,  pre-award  peer-review  requirements  are  separated  into 
two  categories:  competitive  and  noncompetitive  acquisitions.  Pre-award  peer  reviews  for 
competitive  acquisitions  are  required  at  three  different  points:  issuing  the  solicitation, 
requesting  the  final  proposal  revisions,  and  issuing  the  contract  award.  For 
noncompetitive  acquisitions,  the  requirement  is  for  peer  reviews  at  negotiation  and 
contract  award.  For  both  competitive  and  non-competitive,  post- award  peer  reviews 
concentrate  on  adequacy  of  competition,  an  assessment  of  contractor  performance,  and 
how  adequate  the  government  surveillance  plan  of  contractor  performance  has  proven  to 
be  (DPAP,  2013b). 

PGI  201.170  also  defines  the  peer-review  team: 

A  senior  official  designated  by  the  [Office  of  the  Secretary  of  Defense] 

OSD  Office  of  Small  Business  Programs  will  participate  as  a  team 
member  on  peer  reviews  of  services  acquisitions.  Teams  will  include 
civilian  employees  or  military  personnel  external  to  the  department, 
agency,  or  component  that  is  the  subject  of  the  peer  review.  (DPAP, 

2013b) 

This  PGI  highlights  the  documents  that  peer-review  teams  must  be  able  to  access. 
Interestingly,  there  are  also  sections  in  the  PGI  that  read  like  best  practices  for  both  the 
pre-  and  post-award  phases: 

Pre-award  Elements  to  be  Addressed 

•  The  process  was  well  understood  by  both  government  and  Industry; 

•  Source  Selection  was  carried  out  in  accordance  with  the  Source  Selection 
Plan  and  RFP; 

•  The  Source  Selection  Evaluation  Board  (SSEB)  evaluation  was  clearly 
documented; 
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•  The  Source  Selection  Advisory  Council  (SSAC)  advisory  panel 
recommendation  was  clearly  documented; 

•  The  Source  Selection  Authority  (SSA)  decision  was  clearly  derived  from 
the  conduct  of  the  source  selection  process; 

•  All  source  selection  documentation  is  consistent  with  the  Section  M 
evaluation  criteria;  and 

•  The  business  arrangement. 

Post-award  Elements  to  be  Addressed 

•  Contract  performance  in  terms  of  cost,  schedule,  and  requirements; 

•  Use  of  contracting  mechanisms,  including  the  use  of  competition,  the 
contract  structure  and  type,  the  definition  of  contract  requirements,  cost  or 
pricing  methods,  the  award  and  negotiation  of  task  orders,  and 
management  and  oversight  mechanisms; 

•  Contractor’s  use,  management,  and  oversight  of  subcontractors; 

•  Staffing  of  contract  management  and  oversight  functions; 

•  Extent  of  any  pass-throughs,  and  excessive  pass-through  charges  by  the 
contractor  (as  defined  in  section  852  of  the  National  Defense 
Authorization  Act  for  Eiscal  Year  2007,  Public  Eaw  109-364);  and 

•  Steps  taken  to  mitigate  the  risk  that,  as  implemented  and  administered, 
non-personal  services  contracts  may  become  de  facto  personal  services 
contracts. 

The  following  section  discusses  how  corporations  review  procurement  actions 
versus  how  the  DOD  examines  its  procurement  actions  in  the  form  of  the  peer-review 
process  for  contractual  actions. 

E,  CORPORATE  PROCUREMENT  MANAGEMENT,  FEDERAL 

GOVERNMENT  CONTRACTING,  AND  DOD  CONTRACTING:  A  BRIEF 

COMPARISON 

Initially,  a  quick  comparison  of  procurement  in  the  corporate  world  against 
contracting  within  the  federal  government  might  lead  the  uninitiated  to  think  they  were 
similar.  After  all,  both  follow  the  same  six-step  process:  procurement  planning, 
solicitation  planning,  solicitation,  source  selection,  contract  administration,  and  contract 
closeout  or  termination  (Garrett,  2007  p.  21).  Under  more  detailed  inspection,  vast 
differences  within  the  motivation  of  the  parties  involved  and  the  forces  that  shape  the 

process  are  exposed.  Kovacs  (2004)  noted  in  his  work  Enhancing  Procurement  Practices, 
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The  basic  procurement  principles  and  techniques  are  equally  applicable  in 
both  the  public  and  private  sectors  .  .  .  which  means:  with  the  right  quality, 
functionality  and  performance;  under  the  right  terms  and  conditions, 
among  which  costs  and  timely  implementation  usually  are  of  essence;  and, 
with  the  right  long-term  operation  and/or  maintenance  support,  (p.  175) 

The  divergence  begins  with  money  in  the  form  of  funding  and  the  method  by 
which  the  government  sector  and  private  sector  come  by  the  funding. 

Corporations  gain  funding  by  financing  debt  or  by  spending  their  own  profit  they 
have  retained  in  the  corporation.  In  either  instance,  they  are  motivated  by  the  need  and 
desire  to  utilize  those  funds  as  efficiently  as  possible  to  earn  the  maximum  profit  possible 
from  each  dollar  spent.  Thus,  there  is  no  need  for  a  regulatory  requirement  within  the 
overarching  corporate  world  for  a  formal  peer-review  process  of  contracts.  As  long  as 
free  market  forces  are  in  play,  each  corporation  is  driven  by  competition  to  be  as  efficient 
as  possible  (Kovacs,  2004). 

The  chasm  widens  when  considering  business  relationships  and  how  they 
integrate  with  procurement  in  the  corporate  arena  versus  contracting  in  the  government 
sector.  Procurement  actions  outside  of  the  government  are  typically  more  focused  on  the 
business  relationship  and  building  and  maintaining  a  partnership  for  future  ventures. 
These  are  the  incentives,  intangible  and  tangible,  such  as  more  favorable  credit  terms  on 
purchases,  that  drive  corporations  to  follow  through  on  the  whole  procurement  process. 
Hence,  they  are  less  apt  to  neglect  the  contract  administration  and  contract  closeout/ 
termination  phases  of  the  procurement  process  over  federal  contracting  activities. 

Corporations  do  have  methods  for  evaluating,  or  perhaps  more  accurately, 
examining  how  they  are  performing.  Consulting  services  are  often  hired  from  sources 
outside  of  the  corporation  (or  in  cases  of  extremely  large  corporations,  they  may  be 
contained  in-house  under  a  cloistered  division).  Consultants  execute  functions  similar  to 
what  a  peer  review  accomplishes,  in  that  someone  with  an  equal  or  greater  amount  of 
knowledge  on  the  topic  examines  a  subject  (Garrett,  2007).  The  difference  lies  in  the  fact 
that,  with  the  exception  of  extremely  large  contracts,  consultant  actions  typically  focus  on 
a  process,  with  an  expected  end  state  of  further  optimizing  that  process.  Peer  reviews  of 

DOD  contracts  are  conducted  once  Pre  Award  and  once  Post  Award,  as  such  the  peer- 
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review  proeess  is  evaluating  an  end  produet  rather  than  the  proeess  that  delivered  the 
produet. 

There  is  a  tool  that  is  designed  to  help  both  buyers  and  sellers  evaluate  risk  and 
reward  assoeiated  with  proeurements.  “The  Contraet  Management  Risk  and  Opportunity 
Assessments  Tool  (CMROAT)  is  designed  to  help  organizations,  both  buyers  and  sellers, 
assess  the  risk  and  opportunities  assoeiated  with  a  pending/potential  or  aetual  eontraet” 
(Garrett,  2007,  p.  234).  The  CMROAT  has  developed  into  a  tool  utilized  by  both 
government  and  private  entities;  however,  it  foeuses  on  the  risks  and  opportunities 
assoeiated  with  the  business  dealings  of  the  eontraet,  and  has  less  to  do  with  the  strueture 
and  eontent  of  the  eontraet  itself 

Overall,  the  federal  eontraeting  proeess  is  proeess-driven;  if  the  proeess  is 
followed  eorreetly,  then  the  end  produet  is  typieally  in  the  aeeeptable  range. 
Alternatively,  as  noted,  the  eorporate  proeurement  system  is  fueled  by  ineentives.  As  a 
result,  eorporate  review  meehanisms  tend  to  foeus  on  optimizing  a  proeess  for  a  greater 
return  on  investment,  and  the  peer-review  proeess  as  it  applies  to  DOD  eontraetual 
aetions  foeuses  on  fixing  an  output  of  a  proeess  to  bring  it  in  line  with  what  is  deemed 
aeeeptable. 

F.  SUMMARY 

In  the  literature  review,  I  mentioned  several  aspeets  of  the  aequisition  system  and 
how  they  eontributed  to  the  peer-review  requirement.  With  the  implementation  of  the 
DAWIA,  the  DOD  made  signifieant  strides  in  developing  a  professional  aequisition 
workforee.  Negligenee  of  the  workforee  throughout  the  late  1990s  and  into  the  2000s  led 
to  a  shortage  within  the  aequisition  workforee,  followed  by  a  surge  in  aequisition  aetivity 
aeeompanying  the  rapid  inerease  in  overseas  eontingeney  operations.  With  the 
eontraeting  system  thoroughly  stressed,  numerous  examples  of  ineffieieneies,  fraud, 
waste,  and  abuse  were  uneovered.  This  led  to  an  offieial  peer-review  requirement  from 
Congress.  The  aetual  peer-review  proeess  was  developed  based  on  an  existing  pilot 
review  program,  and  expanded  to  meet  the  eongressional  requirement.  The  following 
ehapter  diseusses  the  researeh  methodology. 
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III.  METHODOLOGY 


A,  INTRODUCTION 

This  chapter  addresses  how  I  collected  the  data  and  used  the  analyties.  It  also 
includes  a  summary  of  the  DPAP  database  of  peer-review  results  to  provide  context  for 
the  data  analysis  chapter  of  my  researeh.  The  various  aspects  of  the  DPAP  database  of 
peer-review  results  are  also  described.  The  purpose  of  my  research  was  not  to  provide  an 
exhaustive  answer  to  an  in-depth  research  question;  rather,  the  purpose  was  to  explore  a 
previously  uneharted  research  area  to  examine  if  it  merited  further  study.  The  DPAP 
database  of  peer-review  results  was  broken  down  categorically  by  the  various  data 
elements  of  data  present  and  analyzed  quantitatively  based  on  frequeney  of  oeeurrences 
of  a  particular  data  element. 

B,  DPAP  DATABASE  OF  PEER-REVIEW  RESULTS  OVERVIEW 

The  DPAP  database  of  peer-review  results  resides  in  the  contract  policy  section  of 
the  DPAP  website,  in  both  PDF  and  Excel  versions.  The  purpose  of  the  DPAP  database 
of  peer-review  results  is  to  eollect  the  results  from  peer  reviews  of  contracts  that  met  the 
threshold  ($1  billion  and  above)  for  DOD-level  peer  review.  Aceording  to  the  Peer 
Review  seetion  of  the  DPAP  website: 

The  Peer  Review  program  improves  the  quality  of  the  Department’s 
contraeting  proeesses  by  sharing  lessons  learned  and  best  praetices  and 
ensuring  that  contracting  officers  implement  policy  and  regulations  in  a 
consistent  manner.  The  program  is  implemented  in  accordance  with  DODI 
5000.02  and  DEARS  Part  201.  (DPAP,  2014) 

1.  Summary  of  the  Parts  of  the  DPAP  Database  of  Peer-Review  Results 

The  DPAP  database  of  peer-review  results  itself  is  organized  in  a  simple  format. 
Eigure  6  shows  an  example  page  from  the  DPAP  database  of  peer-review  results  as  of 
Deeember  20,  2013. 
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Figure  6.  Excerpt  from  DPAP  Database  for  OSD-Level  Peer  Reviews 

(from  DPAP,  2013a) 


OSO  Level  Peer  Reviews 

Best  Practices,  Lessons  Learned,  and  Recommendations  (as  of  December  20, 2013) 


Catctory 

irypc  of  Contract 

Feedback 

Review  Phase 

Type  of  Feedback 

Incentive  and 

Award  Fee 

logistical  Services 
(Competitive) 

Solicitation  contained  provision  for  an  award-term.  Peer  Review  Team  (PRT)  recommended  adding 
language  to  indicate  that  an  award  of  'Excellent*  does  not  guarantee  exercise  of  an  award  term  optiori  liy 
the  Government. 

Phase  1 

Recommendation 

Incentive  and 

Award  Fee 

( xmpetitive  Services 
Contraa 

The  planned  contract  structure  provides  for  a  Cost-Plus-Incentive-fee  (CPIF)  arrangement  with  incentives 
only  relating  to  cost.  The  Peer  Review  Team  (PRT)  recommends  incentivizing  schedule  and/or  technicail 
performance  as  well  as  cost  The  negotiated  incentive  arrangement  could  lake  into  account  the 
contractor's  protected  cash  flow. 

Phase  2 

Recommendation 

Incentive  and 

Award  Fee 

IVon-Competitive  Services 
Contract 

Compared  to  other  types  of  contractors.  Federally  Funded  Research  &  Development  Centers  (FFRDCs) 
have  unique  fee  arrangements,  which  are  discussed  in  DFARS  21S.404.7S,  Fee  Requirements,  typically 
referred  to  as  *fee-for-need.‘  DFARS  states  that  FFRDCs  may  be  provided  no  fee  and  that  the  decision  to 
pay  fee  rest  solely  with  the  contracting  offer. 

Phase  1 

Recommendation 

Incentive  and 

Award  Fee 

Competitive  Multiple  Award 
(Combination  of  firm-fixed 
(Mice  and  cost  type  line 
items) 

Award  Fee  Plan  (AFP)  does  not  reflect  the  best  business  arrangement  for  the  government  in  the  following 
areas:  a)  Roll-Over  of  award  fee  amounts:  b)  Provisional  fees.  Therefore,  the  acquisition  team  was 
encouraged  to  consider  developing  cost  and  performance  incentives  in  lieu  of  award  fees  in  accordance 
with  DoD  policy. 

Phase  1 

Recommendation 

Incentive  and 

Award  Fee 

Competitive  Multiple  Award 
liervices  Contract 

As  drafted,  offerors  wiR  propose  a  fee  pool  percentage  and  the  evaluation  team/Procuring  Contractmg; 

Officer  (PCO)  must  determine  whether  or  not  the  offer  is  'balanced.'  The  Peer  Review  T earn  (PRT) 
suggested  defining  the  fee  pool  percentage  and  not  allowing  the  offerors  to  propose  a  percentage.  This 
ensures  there  is  an  adequate  pool  and  simplifies  the  evaluation  of  proposals  by  eliminating  the  need  to 
evaluate  'balance.' 

Phase  1 

Recommendation 

Incentive  and 

Award  Fee 

IVon-Competitive  Weapon 
liystem 

The  Peer  Review  Team  (PRT)  recommended  the  contracting  officer  ensure  that  cost  is  not  considered  in 
both  incentive  fee  and  award  fee,  that  maximum  use  of  objective  criteria  be  used  to  develop  award  fe(t 
criteria  whenever  possible,  and  that  performance  incentives  are  duplicated  in  the  award  fee  criteria. 
Although  the  draft  award  fee  plan  incorporates  some  objective  measure,  there  is  still  much  vagueness,  in 
the  wording  and  more  measures  of  processes  than  outcomes. 

Phase  2 

Recommendation 

Incentive  and 

Award  Fee 

Non-Competitive  Weapon 

iWstem 

The  amount  or  percent  allocated  for  each  performance  incentive  event  should  be  gradual  in  nature,  so  the 
fees  are  not  front-loaded  and  ample  lee  is  left  through  contract  completion.  In  addition,  clear  definition  of 
success  for  all  events  must  be  communicated  to  the  contractor.  For  example,  the  definition  lor  exit  and 
entrance  criteria  and  what  comprises  an  IPR  (In  Process  Review)  was  not  noted  in  the  attachment  referred 
to  in  the  solicitation.  Likewise,  we  recommend  the  Contracting  Officer  consider  whether  performing  an  IBR 
with  180  days,  which  is  required  by  DFARS  clause  2S2.234-7002(e),  should  be  part  of  any  performance 
incentive.  If  it  is,  the  resulting  fee  percent  allocated  to  it,  should  be  minimal  and  the  criteria  lor  successful 
completion  clearly  spelled  out  in  the  resulting  contract  (as  currently,  the  information  relating  to  the  IBR  is 
scant). 

Phase  2 

Recommendation 

Each  row  consists  of  an  entry  for  a  particular  data  element  of  a  eontract,  and 
the  columns  contain  the  appropriate  type  of  data  related  to  that  eontraet  entry.  The 
DPAP  database  of  peer-review  results  has  been  anonymized,  so  there  is  no  way  to 
diseern  how  many  entries  apply  to  the  same  eontract.  The  Category  eolumn  lists  the 
category  the  peer-review  eomment  falls  within.  There  are  eight  Categories:  Ineentive  and 
Award  Fee,  Market  Researeh,  Peer  Review,  Post-award  Administration,  Prieing, 
Requirements/Performanee  Work  Statement/Seope  of  Work,  Souree  Seleetion,  and 
Terms  and  Conditions.  The  Types  of  Contraets  eolumn  spans  a  staggering  31  different 
types,  eonsisting  of  competitive  and  non-eompetitive,  services,  goods,  and  construetion 
eontraets. 

The  Feedbaek  eolumn  provides  a  simple  narrative  eomment  regarding  the  opinion 
of  the  peer-review  team.  The  eolumn  entitled  Review  Phase  illustrates  the  phase  of  the 
eontracting  proeess.  There  are  eurrently  six  phases:  Phase  1,  Phase  2,  Phase  2&3,  Phase 
3,  Phase  4,  and  Post  Award  1.  The  final  eolumn  relates  to  the  Type  of  Feedback  provided, 
and  eonsists  of  three  eategories:  Recommendation,  Best  Practiee,  and  Fessons  Fearned. 
Figure  6  summarizes  the  different  database  fields. 

2.  Phases  of  the  Contracting  Process  versus  DPAP  Peer-Review  Phases 

The  DPAP  database  of  peer-review  results  provides  no  explanation  about  the 
timing  of  the  Review  Phases  used,  nor  does  the  Frequently  Asked  Questions  or  Standard 
Operating  Proeedures  for  peer  reviews  on  DPAP’s  website.  Further  researeh  uneovered 
an  explanation  of  four  phases  in  the  GAO  (2010)  report  Status  ofDOD’s  Implementation 
of  Independent  Management  Reviews  for  Services  Acquisitions.  Phase  1  is  prior  to  the 
issuanee  of  the  solieitation.  The  doeuments  reviewed  eonsist  of  the  performanee  work 
statement,  quality  assuranee  surveillanee  plan,  request  for  proposal,  and  souree  seleetion 
plan.  Phase  2  is  prior  to  the  request  for  final  proposal  revisions.  The  doeuments  reviewed 
are  instruetions  for  proposals  and  proposal  evaluation  criteria,  source  seleetion  evaluation 
guide,  source  seleetion  plan,  and  evaluations  of  eontraetor  proposals.  Phase  3  is  prior  to 
eontraet  award.  The  doeuments  reviewed  eonsist  of  the  proposal-analysis  report  and 
seleetion-deeision  doeument.  Phase  4  is  the  post-award  review.  The  doeuments  reviewed 
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consist  of  any  documentation  related  to  the  program,  sueh  as  task  orders,  award-fee  plan, 
and  performanee  assessments  (GAO,  2010,  p.  6).  Figure  7  highlights  the  eomparison 
between  the  DPAP  Peer  Review  Phases  and  phases  of  the  eontraeting  proeess. 


DPAP  Peer 

ReviewPhases 

1)  Prior  to  issuance  of  the  solicitation 

(No  comparable 
step) 

2)  Prior  to  request  for  final  proposal 
revisions  &  3)  Prior  to  contract  award 

4)  Post-award  review 

Contracting 
Process  Phases 

1)  ProcurenKnt 
Planning 

2)  Solicitation 
Planning 

3)  Solicitation 

4)  Source  Selection 

5)  Contract 
Administration 

6)  Contract 
Closeoit 

Figure  7.  DPAP  Peer-Review  Phases  Compared  to  the 
Contraeting  Process  Phases 


The  DPAP  database  of  peer-reviews  Review  Phases  aetually  eontain  Phases  1 
through  4,  and  an  additional  Post-award  1  phase.  Analyzing  the  Narrative  feedbaek  seems 
to  indieate  that  in  praetiee,  the  Phase  4  review  deseribed  in  the  aforementioned  GAO 
(2010)  report  oeeurs  shortly  after  eontraet  award,  and  the  Post-award  1  review  oeeurs 
well  after  contraetor  performanee  is  underway. 

C.  TYPES  OF  DATA  ANALYSIS 

1,  DPAP  Database  of  Peer-Review  Results  Data  Element  Fields 

Within  the  DPAP  database  of  peer-review  results,  there  are  288  reeords.  Eaeh 
data  element  was  subdivided  within  the  spreadsheet  to  allow  for  the  identifieation  of  the 
frequeney  of  oeeurrenee  for  eaeh  data  element.  This  data  was  then  used  to  identify  trends 
within  the  DPAP  database  of  peer-review  results.  Deseriptive  analysis  of  the  frequeney  of 
observations  for  eaeh  eategory  of  feedbaek  was  eondueted. 

The  eight  Categories,  31  Types  of  Contraets,  six  Review  Phases,  and  three  Types 
of  Feedbaek  were  individually  extraeted  from  the  DPAP  database  of  peer-review  results. 
The  results  of  eaeh  data  element  was  tallied  and  reeorded  and  then  analyzed  by  frequeney 
of  oeeurrenee  against  the  other  data  elements,  with  the  goal  of  identifying  overarehing 
trends  in  the  data  elements. 

Sinee  the  preponderanee  of  the  data  elements  were  eontained  in  Reeommendation 
Type  of  Feedbaek,  the  Category,  Type  of  Contraet,  and  Review  Phase  data  elements  were 
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further  analyzed  with  an  eye  toward  identifying  how  mueh  Reeommendation  Type  of 
Feedbaek  appeared  in  eaeh  of  those  data  elements.  To  provide  additional  granularity, 
oeeurrenees  of  Reeommendations  were  expressed  as  pereent  of  the  total  oeeurrences  for 
both  Categories  and  Contraet  Type  data  elements. 

Competency  gaps  identified  in  the  DOD  Contracting  Workforce  Competency 
Assessment  Final  Report  (DPAP,  2007)  were  cross-referenced  against  the  DPAP 
database  of  peer-review  results  to  identify  any  areas  of  overlap  between  the  two.  Once 
areas  of  overlap  had  been  identified,  occurrences  of  Recommendation  Type  of  Feedback 
for  the  overlap  areas  was  reviewed  to  see  if  the  identified  competency  gaps  had  increased 
occurrences  of  Recommendations. 

2,  Narrative  Feedback  Themes 

The  Feedback  section  for  each  entry  was  also  examined  to  identify  any  common 
themes  within  the  feedback  provided.  The  primary  goal  for  examining  the  content  of  the 
Feedback  section  was  to  identify  apparent  incongruities  between  listed  categories,  such  as 
feedback  type,  and  the  actual  narrative  posted  within  the  Feedback  column.  My 
examination  is  concerned  with  identifying  irregularities  with  policy  and  regulations. 

D,  SUMMARY 

The  Methodology  chapter  provided  a  brief  overview  of  the  DPAP  database  of 
peer-review  results,  and  how  it  was  subdivide  for  analysis.  The  Review  Phases  were 
compared  to  the  phases  of  the  contracting  process.  How  the  narrative  Feedback  themes 
were  reviewed  was  also  addressed.  The  next  chapter.  Chapter  IV,  presents  the  results  of 
the  data  analysis. 
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IV.  DATA  ANALYSIS 


A,  INTRODUCTION 

The  previous  chapter  discussed  how  the  data  was  analyzed.  This  section  focuses 
on  trends  within  the  DPAP  database  of  peer-review  results  by  various  data  elements  of 
the  data.  Frequencies  of  occurrences  by  Category,  Type  of  Contract,  Review  Phase,  and 
Type  of  Feedback  are  examined.  The  Recommendation  Type  of  Feedback  receives 
additional  review  because  of  its  predominance  of  occurrences  within  the  DPAP  database 
of  peer-review  results.  Relationships  between  the  DOD  Contracting  Competency  Model 
(CCM)  and  trends  within  the  DPAP  database  of  peer-review  results  are  compared. 
Finally,  themes  within  the  narrative  feedback  section  are  also  investigated. 

B,  DESCRIPTIVE  STATISTICS 

The  various  data  elements  within  the  DPAP  database  of  peer-review  results  noted 
in  the  methodology  chapter  were  analyzed  for  frequency  of  occurrence.  The  importance 
of  breaking  down  the  DPAP  database  of  peer-review  results  in  this  manner  lies  in  the  fact 
that  it  allows  contracting  officers  and  policymakers  to  identify  areas  where  systemic 
issues  within  the  contracting  process  persist.  The  data  on  frequencies  of  occurrences  and 
the  underlying  trends  can  then  be  used  to  target  the  development  of  educational  and 
training  programs  to  address  problem  areas.  A  process  as  complex  as  developing  high- 
dollar  defense  contracts  may  well  never  be  without  faults,  but  previous  deficiencies  need 
to  be  addressed  to  avoid  future  duplication  of  effort. 

1.  Category 

Working  across  the  DPAP  database  of  peer-review  results,  the  first  data  element 
encountered  is  the  Category  column.  The  largest  category  was  the  source  selection 
category  at  115  of  288  observations,  which  should  not  be  surprising  given  two  facts: 
(1)  the  emphasis  that  is  placed  on  the  source  selection  process  within  DOD  contracting; 
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and  (2)  the  number  of  protests  that  originate  in  the  source  selection  process, 
stemming  from  allegations  of  issues  such  as  unequal  application  of  evaluation  criteria, 
unfair  negotiation  and  discussion  practices,  and  poor  use  of  Best  Value  trade-offs  and 
Lowest  Price  Technically  Acceptable  criteria  for  contract  award  (White,  2012).  Figure  8 
represents  the  occurrences  of  all  the  category  types  represented  in  the  DPAP  database  of 
peer-review  results. 


Terms  &  Conditions 

Source  Selection 
Requirements/PWS/SOW 
Pricing 
Post  Award  Admin 

Peer  Review 

Market  Research 

Incentive  and  Award  Fee 


Occurrences  by  Category  Type 


0  20  40  60  80  100  120 


■  Occurrences 


Figure  8.  Frequency  Distribution  of  Category  Type  (after  DPAP,  2013a) 
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The  second  cluster  of  occurrences  centered  on  Terms  and  Conditions,  Pricing,  and 
Post  Award  Administration.  It  is  notable  that  pricing,  while  the  second  largest  category, 
is  still  a  distant  second  with  46  occurrences.  Given  issues  that  seem  to  pervade  the  pricing 
realm  within  contracting,  it  would  seem  that  this  area  would  receive  more  attention. 
Bringing  the  point  into  sharper  focus  is  the  consideration  that  this  data  represents  high- 
do  liar  contracts  valued  at  $1  billion  and  above,  and  that  high  value  defense  procurements 
often  have  significant  cost  overruns  exceeding  20%  of  the  original  cost  estimate  (GAO, 
2014). 


2,  Type  of  Contract 

The  next  data  element  of  the  DPAP  database  of  peer-review  results  is  the  Type  of 
Contract  column.  The  contract  types  listed  comprise  a  staggering  3 1  types  of  contracts. 
The  most  frequently  occurring  type  is  the  Competitive  Multiple  Award  Services  Contract 
at  59  occurrences,  followed  by  the  Competitive  Services  Contracts  and  Competitive 
Multiple  Award  indefinite  delivery  indefinite  quantity  (IDIQ)  at  39  and  20  occurrences, 
respectively.  Figure  9  shows  the  various  types  of  contracts  listed  and  the  number  of  times 
the  occurrence  was  observed. 
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Occurrences  by  Contract  Type 


Weapon  System  Technology  Demonstration 
Weapon  System  Production  Lot  Buy  (Sole  Source) 
Weapon  System  Development  Sole  Source 
Weapon  System  Development 
Sole  Source  Procurement  of  Services 
Non  Competitive  Weapon  System  Development 
Non  Competitive  Weapon  System 
Non  Competitive  Supplies  Contract 
Non  Competitive  Services  Contract 
Non  Competitive  Multiple  Award  Services  Contract 
Multiple  Award  Contract  for  R&D 
Multiple  Award  Construction  Contract 
Logistical  Services  (Competitive) 
Design/Build  Construction  Contract 
Competitive  Weapons  System  Development 
Competitive  Supplies  Contract 
Competitive  Services  Contract 
Competitive  Procurement  of  Supplies 
Competitive  Procurement  of  Services 
Competitive  Multiple  Award  IDIQ  Supplies  Contract 
Competitive  Multiple  Award  IDIQ 
Competitive  Multiple  Award  Services  Contract 
Competitive  Multiple  Award  IDIQ  Services 
Competitive  Multiple  Award  IDIQ  R&D  Contract 
Competitive  Multiple  Award  (Combination  Multiple. 

Competitive  IQC 
Competitive  IDIQ  Services  Contract 
Competitive  IDIQ  for  IT  Products 
Competitive  IDIQ 
Competitive  Design/Build  FPIF 
Commodity  (Competitive) 


■  Occurrences 


Figure  9.  Frequency  Distribution  of  Contract  Type  (after  DPAP,  2013a) 
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Admittedly,  the  terminology  of  some  of  the  listed  contract  types  could  be 
combined,  such  as  Competitive  Supplies  Contract  and  Competitive  Procurement  of 
Supplies.  The  issues  associated  with  having  such  a  broad  scope  of  contract  types  listed  is 
addressed  in  a  subsequent  section  of  this  analysis  because  this  particular  part  of  the 
DPAP  database  of  peer-review  results  is  one  of  two  that  could  benefit  the  most  from 
refinement.  The  key  takeaway  from  this  section  of  data  appears  to  be  that,  overall,  the 
DOD  is  peer  reviewing  more  contracts  for  services  than  anything  else.  Service  contracts 
comprise  over  65%,  or  189  occurrences,  of  the  Contract  Type  entries.  This  is  not 
surprising,  considering  the  amount  of  services  contracted  for  within  the  DOD,  but  what 
is  worth  considering  is  that  those  189  occurrences  represent  a  dollar  value  of  at  least 
$1  billion  each  (GAO,  2008). 

3,  Review  Phase 

The  next  data  element  to  address  is  the  Review  Phase  column.  Again,  one  area 
contains  the  majority  of  the  occurrences;  this  time  they  fall  into  Phase  1,  with  a  total  of 
157.  Phase  2  follows  with  64,  and  Phase  3  and  Post-award  1  are  nearly  tied  at  28  and 
27  occurrences.  Figure  10  shows  the  occurrences  by  review  phase. 


Occurrences  by  Review  Phase 


I  Occurrences 


Figure  10.  Frequency  Distribution  of  Review  Phase  (after  DPAP,  2013a) 
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The  trends  demonstrated  within  the  occurrences  by  review  phase  would  seem  to 
indicate  that  learning  is  taking  place  as  the  contract  process  moves  along.  It  also  seems  to 
break  itself  naturally  between  Phase  4  and  Post-award  1,  when  the  contract  would  have 
been  awarded  at  the  conclusion  of  the  source  selection  process  and  contract  performance 
had  begun.  This  action  would  move  the  contract  into  the  contract  administration  portion 
of  the  contracting  process.  Since  contract  administration  represents  a  very  different  phase 
of  the  contracting  process  from  procurement  planning,  solicitation  planning,  solicitation, 
and  source  selection,  it  follows  that  additional  learning  would  begin  to  occur  in  the  Post¬ 
award  1  phase. 

4,  Type  of  F eedback 

The  final  quantifiable  data  element  of  the  DPAP  database  of  peer-review  results  is 
the  Type  of  Feedback  column.  This  column  seeks  to  assign  a  type  to  the  narrative 
feedback  column.  Consistent  with  the  other  DPAP  database  of  peer-review  results 
categories,  one  category  heavily  outweighed  the  others  with  occurrences. 
Recommendation  contained  190  occurrences,  with  Best  Practice  containing  68  and 
Lessons  Learned  amassing  30  occurrences.  Figure  11  demonstrates  the  occurrences  by 
Type  of  Feedback. 


Occurrences  by  Feedback  Type 


Lessons  Learned 


Best  Practice 


Recommendation 


■  Occurrences 


Figure  1 1 .  Frequency  Distribution  of  Type  of  Feedback  (after  DPAP,  2013a) 
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The  type  of  feedback  aspect  is  the  other  data  element  of  the  DPAP  database  of 
peer-review  results  that  could  benefit  from  additional  refinement.  The  Lessons  Learned 
Type  of  Feedback  ostensibly  provides  a  way  to  communicate  an  item  of  information  the 
peer-review  team  was  not  aware  of  to  the  readership  of  the  DPAP  database  of  peer- 
review  results.  In  reading  the  narrative,  it  often  becomes  difficult  to  distinguish  between  a 
Lesson  Learned  and  a  Recommendation.  Best  Practices  are  just  that — a  best  practice  that 
the  peer-review  team  observed  from  the  organization  or  contract  that  the  team  was 
reviewing;  it  exists  as  a  way  to  propagate  those  practices  back  out  to  the  contracting 
workforce.  With  only  three  types  of  feedback  categories,  and  since  most  of  the 
occurrences  are  in  a  single  category,  it  is  difficult  to  ascertain  anything  quantitatively 
unless  the  recommendation  category  is  delved  into  further,  which  occurs  in  the  next 
section. 


5.  Recommendation  Category 

The  Recommendation  category  contained  190  out  of  288  occurrences  regarding 
feedback  type.  That  comprises  65.97%  of  the  total  feedback  types.  It  is  also  the  single 
largest  density  of  occurrences  anywhere  in  the  DPAP  database  of  peer-review  results.  As 
such,  further  analysis  was  conducted. 

Examining  the  phases  revealed  that  most  Recommendations  were  concentrated  in 
Phase  1  reviews.  Phase  1  contained  130  Recommendation  occurrences.  Phase  2  was  a 
distant  second  at  38  occurrences,  with  12  occurrences  in  Phase  3,  none  in  Phase  4,  and  1 1 
in  Post-award  1.  Figure  12  elucidates  the  recommendation  occurrences  by  phase. 
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Recommendation  Occurrences  By  Phase 
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Figure  12.  Frequency  Distribution  of  Recommendations  by  Phase 

(after  DPAP,  2013a) 

The  findings  are  consistent  with  the  idea  that  learning  is  taking  place  throughout 
the  contracting  process,  and  organizations  that  are  subject  to  formal  review  by  a  peer- 
review  team  are  applying  the  observations  of  the  peer-review  team.  It  again  demonstrates 
the  break  between  contract  award  and  contract  performance,  illustrated  by  the  uptick 
in  Recommendations  between  the  Phase  4  Review  Phase  and  the  Post-award  1  Review 
Phase. 

Next,  the  number  of  occurrences  of  recommendations  within  each  Category  was 
examined.  The  categories  were  Terms  &  Conditions,  Source  Selection,  Requirements/ 
PWS/SOW,  Pricing,  Post  Award  Admin,  and  Incentive  and  Award  Fee.  Figure  13 
exhibits  the  number  of  occurrences  of  Recommendations  within  each  Category. 
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Occurrences  of  Recommendation 
Within  Categories 
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Figure  13.  Frequency  Distribution  of  Recommendations  within  Categories 

(after  DPAP,  2013a) 


Another  important  way  to  view  this  data  is  by  percentage  of  Recommendations 
within  each  Category. 


Categories 

Total 

Occurrences 

Occurrences  of 
Recommendation 

Percent  of 
Recommendation 

Incentive  and  Award 
Fee 

23 

20 

86.96  % 

Market  Research 

4 

0 

0% 

Peer  Review 

1 

0 

0% 

Post-award  Admin 

35 

9 

25.71  % 

Pricing 

46 

41 

89.13  % 

Requirements/PW  S/ 
SOW 

24 

17 

70.83  % 

Source  Selection 

115 

66 

57.39  % 

Terms  &  Conditions 

40 

37 

92.5  % 

Total 

288 

190 

65.97  % 

Table  1.  Percentage  of  Recommendations  by  Category 
(after  DPAP,  2013a) 
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Table  1  lists  each  of  the  Category  types  in  the  DPAP  database  of  peer-review 
results.  Total  Occurrences  is  the  number  of  times  that  Category  appears  within  the 
database.  Assuming  that  the  priority  of  the  DPAP  database  of  peer-review  results 
is  to  capture  issues  with  solicitations  and  contracts  to  assist  in  informing  the  DOD 
contracting  workforce,  then  the  third  column  of  the  above  chart  is  the  most  important 
cross  section  of  the  data  to  consider.  It  details  the  number  of  Recommendations  per 
Category  within  the  DPAP  database  of  peer-review  results.  Consider  the  first  row  of 
Table  1;  it  expounds  on  the  Incentive  and  Award  Fee  Category.  That  Category  appeared 
23  times  in  the  DPAP  database  of  peer-review  results;  of  its  23  appearances,  20  of  those 
constituted  Recommendation  Feedback  Type,  with  86.96%  of  the  occurrences  being 
Recommendations.  The  percentage  column  is  included  simply  to  help  visualize  the 
Recommendation  occurrences  by  category.  This  provides  a  more  precise  picture  within 
the  DPAP  database  of  peer-review  results  as  to  where  peer-review  teams  are  uncovering 
issues.  If  one  only  examined  the  occurrences  of  Recommendations,  the  conclusion  might 
be  that  20  Recommendations  might  not  be  significant,  but  once  the  dimension  is  added 
that  20  is  out  of  23,  and  86.96%  of  the  Feedback  Type  for  that  Category  is  a 
Recommendation,  then  there  may  be  an  issue  within  Incentive  and  Award  Fees  that  needs 
to  be  addressed.  By  scrutinizing  the  data  in  that  manner,  it  would  seem  the  areas  of 
concern  that  should  merit  additional  attention  are  Incentive  and  Award  Fee,  Pricing, 
Requirements/PWS/SOW,  and  Terms  &  Conditions.  Curiously  enough,  while  Source 
Selection  as  a  Category  is  the  most  frequently  occurring  Category  within  the  DPAP 
database  of  peer-review  results,  only  57.39%  of  its  appearances  consist  of 
Recommendations,  with  the  rest  comprised  by  Best  Practices  and  Lessons  Learned. 

The  final  portion  of  the  DPAP  database  of  peer-review  results  to  be  examined  in 
relation  to  Recommendations  is  by  Contract  Type.  Figure  14  lists  the  number  of 
occurrences  of  Recommendations  within  each  listed  Contract  Type. 
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Occurrences  of  Recommendation  Type  Feedback  by 
Contract  Type 


Weapon  System  Technology  Demonstration 
Weapon  System  Production  Lot  Buy  (Sole  Source) 
Weapon  System  Development  Sole  Source 
Weapon  System  Development 
Sole  Source  Procurement  of  Services 
Non  Competitive  Weapon  System  Development 
Non  Competitive  Weapon  System 
Non  Competitive  Supplies  Contract 
Non  Competitive  Services  Contract 
Non  Competitive  Multiple  Award  Services  Contract 
Multiple  Award  Contract  for  R&D 
Multiple  Award  Construction  Contract 
Logistical  Services  (Competitive) 
Design/Build  Construction  Contract 
Competitive  Weapons  System  Development 
Competitive  Supplies  Contract 
Competitive  Services  Contract 
Competitive  Procurement  of  Supplies 
Competitive  Procurement  of  Services 
Competitive  Multiple  Award  IDIQ  Supplies  Contract 
Competitive  Multiple  Award  IDIQ 
Competitive  Multiple  Award  Services  Contract 
Competitive  Multiple  Award  IDIQ  Services 
Competitive  Multiple  Award  IDIQ  R&D  Contract 
Competitive  Multiple  Award  (Combination  Multiple  Award- 

Competitive  IQC 
Competitive  IDIQ  Services  Contract 
Competitive  IDIQ  for  IT  Products 
Competitive  IDIQ 
Competitive  Design/Build  FPIF 
Commodity  (Competitive) 


13 


41 


0 


10 


20 


30 


40 


50 


I  Occurrences  of  Recommendation 


Figure  14.  Frequency  Distribution  of  Recommendations  by  Contract  Type 

(after  DPAP,  2013a) 
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Similarly  to  the  way  the  data  for  reeommendations  within  eaeh  eategory  was 
disseeted,  Table  2  provides  a  breakdown  of  pereentage  of  eaeh  eontract  type  that  returned 
a  reeommendation. 


Contract  Type 

Total 

Occurrences  of 
Recommendation 

Percentage  of 
Recommendation 

Commodity  (Competitive) 

4 

3 

75% 

Competitive  Design/Build  FPIF 

2 

2 

100  % 

Competitive  IDIQ 

1 

1 

100  % 

Competitive  IDIQ  for  IT  Products 

4 

4 

100  % 

Competitive  IDIQ  Services  Contract 

6 

2 

33.33  % 

Competitive  IQC 

1 

1 

100  % 

Competitive  Multiple  Award  (Combination 

7 

7 

100  % 

Multiple  Award  FFP  and  Cost  type  line  items) 

Competitive  Multiple  Award  IDIQ  R&D 

1 

0 

0% 

Contract 

Competitive  Multiple  Award  IDIQ  Services 

22 

3 

13.64% 

Competitive  Multiple  Award  Services 

59 

41 

69.49  % 

Contract 

Competitive  Multiple  Award  IDIQ 

5 

2 

40% 

Competitive  Multiple  Award  IDIQ  Supplies 

2 

1 

50% 

Contract 

Competitive  Procurement  of  Services 

20 

9 

45% 

Competitive  Procurement  of  Supplies 

14 

9 

64.29  % 

Competitive  Services  Contract 

39 

16 

41.03% 

Competitive  Supplies  Contract 

1 

1 

100  % 

Competitive  Weapons  System  Development 

16 

14 

87.5  % 

Design/Build  Construction  Contract 

3 

3 

100  % 

Logistical  Services  (Competitive) 

10 

7 

70% 

Multiple  Award  Construction  Contract 

2 

2 

100  % 

Multiple  Award  Contract  for  R&D 

6 

5 

83.33  % 

Non-Competitive  Multiple  Award  Services 

11 

11 

100  % 

Contract 

Non-Competitive  Services  Contract 

4 

4 

100  % 

Non-Competitive  Supplies  Contract 

10 

8 

80% 

Non-Competitive  Weapon  System 

8 

8 

100  % 

Non-Competitive  Weapon  System 

6 

5 

83.33  % 

Development 

Sole-Source  Procurement  of  Services 

1 

1 

100  % 

Weapon  System  Development 

5 

4 

80% 

Weapon  System  Development  Sole  Source 

1 

0 

0% 

Weapon  System  Production  Lot  Buy  (Sole 

3 

3 

100  % 

Source) 

Weapon  System  Technology  Demonstration 

14 

13 

92.86  % 

Total 

288 

190 

65.97  % 

Table  2.  Percentage  of  Recommendations  by  Contract  Type  (after 

DPAP,  2013a) 
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The  variety  of  eontraet  types  listed  makes  it  diffieult  to  pinpoint  precise  issues 
within  the  Contract  Type  data  element.  If  we  aggregate  the  Service  type  contracts,  then 
the  rate  of  Recommendations  is  49.47%.  This  does  not  represent  an  overwhelming  rate 
given  the  concentration  of  Recommendation  Type  Feedback  that  some  data  elements  in 
the  DPAP  database  of  peer-review  results  have  experienced.  Scanning  Table  2  would 
seem  to  indicate  that  the  Contract  Type  data  element  is  garnering  more  than  its  fair  share 
of  Recommendations;  indeed,  rates  upward  of  70%  seem  prevalent.  Two  Contract  Type 
data  elements  are  drastically  pulling  the  average  down.  Competitive  Multiple  Award 
IDIQ  Services  with  three  Recommendations  in  22  occurrences,  and  Competitive 
Procurement  of  Services  with  nine  Recommendations  in  20  occurrences.  With  the  wide 
range  of  Contract  Types  employed  in  this  data  element,  it  will  take  more  entries  in  the 
DPAP  database  of  peer-review  results  to  draw  meaningful  conclusions  about  where 
issues  with  Contract  Type  exist.  The  next  section  will  address  comparisons  between  the 
Contracting  Core  Competencies  and  the  DPAP  database  of  peer-review  results. 

C.  CONTRACTING  CORE  COMPETENCIES  COMPARISONS 

The  DOD  CCM  was  developed  in  2007  after  an  exhaustive  survey  was  conducted 
across  multiple  DOD  contracting  organizations.  The  survey  asked  both  employees  and 
supervisors  to  assess  critical  areas  within  the  contracting  process  they  worked  on,  and 
also  asked  them  to  rate  themselves  from  a  proficiency  standpoint.  Subject  matter  experts 
from  the  contracting  career  field  took  a  list  of  27  business  and  professional  competencies 
and  were  asked  to  rank  them  as  to  how  effective  they  were  in  relation  to  job  performance. 
Ten  competencies  were  identified  as  the  most  important  for  work  performance.  From 
that,  the  CCM  was  developed,  consisting  of  12  units  of  competence,  28  technical 
competencies,  and  10  professional  competencies  (DPAP,  2007).  Table  3  represents  the 
CCM. 
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DoD  Contracting  Competency  Model 
established  March  2007 


Pre-Award  and  Award 

Determmawn  ot  How  Best  To  Satisfy  Reguiremens 

Considers  Sooo-econorTK  Requirements 

Promote  Compeaon 

Source  Seiecson  =*anfw)g 

Soidcation  of  Offers 

Responsfe«y  Deterawaaon 

B«1  Evaluation  (Sealed  Biddvig) 

Proposal  Evaluation  (Contracting  by  Negooaaon) 

Source  SelecJon 

Contraa  Award 

Process  Protests 

Develop  and/or  Negooaoe  Posoons 

Jusifcaton  of  Other  than  Pti  and  Open  Competnon 

Terms  and  Cond«ons 

Preparation  and  Negoaaaon 

Advanced  Cost  and/or  Prce  Analyse 

Advanced  Cost  and/or  Price  Analysts 

Contract  Adm«stra#on 

Inoation  of  work 

Contract  Perfcrmance  Management 

Issue  Changes  and  Moddcatons 

Approve  Payment  Requests 

CJose-out  Contracts 

Smal  Busmess/Sodo-Eccmnc  Programs 

Addressmg  Smal  Busmess  Concerns 

Negottaie  Forward  Pnong  Rate  Agreements 
and  Admrvser  Cost  Accoonanq  Sandards 

Negotiate  Forward  Pnong  Rate  Agreements 
and  Admmister  Cost  Accounonq  Sandards 

Contract  Temwiaoon 

Contraa  Temwaaon 

Procurement  Pofccy 

Procurement  .Analysis 

Other  Competencies 

E-Busmess  and  Aucmated  Tools 

Activiy  Program  Coordmasor  for  Purchase  Card 
Construdon/Archiea  and  Engmeenng 

Coniracong  n  a  Conongent  and/or  Combat  Environment 

Contracting  n  a  Comngent  and/or  Combat  Environment 

Professional  Competency 

Probteffl  Solving 

Cusomer  Service 

Oral  Commumcaoon 

'Artten  Commumcaaon 

Interpersonal  SUb 

Decisiveness 

Technical  CredibSy 

Fiexjbicy 

Resience 

Accoontab*y 

Table  3.  DOD  Contracting  Competency  Model  (from  DPAP,  2007) 
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The  final  report  detailing  the  survey  results  and  providing  the  model  was  entitled 
Department  of  Defense  Contracting  Workforce  Competency  Assessment  Final  Report 
(DPAP,  2007).  Included  in  the  report  were  high-risk  areas  within  the  contracting 
workforce  that  had  been  identified  through  the  survey  process.  These  high-risk  areas 
were  described  as  “Competency  Gaps.”  The  following  is  a  list  of  the  areas  established  as 
Competency  Gaps: 

•  Fundamental  contracting  skills  across  entry  and  journey  levels  of  the 
Contracting  Workforce  and  currency,  breadth,  and  depth  of  knowledge 
across  journey  and  senior  levels; 

•  The  Source  Selection  Process 

•  Cost  and  Price  Analysis 

•  Contract  Performance  Management 

•  Integrated  Acquisition  Skills.  (DPAP,  2007) 

From  these  Competency  Gaps,  a  few  areas  from  the  DPAP  database  of  peer- 
review  results  can  be  cross-referenced.  Within  the  DOD  CCM,  the  source  selection 
process  is  broken  down  into  the  following  subparts:  source  selection  planning,  source 
selection,  proposal  evaluation,  and  contract  award.  Referencing  the  previous  section  on 
the  Recommendation  Feedback  Type  breakdown  by  phase,  the  greatest  frequency  of 
Recommendations  by  phase  was  in  Phase  1.  Phase  1  would  correlate  with  source- 
selection  planning  within  the  CCM,  but  since  the  Department  of  Defense  Contracting 
Workforce  Competency  Assessment  Final  Report  (DPAP,  2007)  did  not  break  out  any  of 
the  sub-areas  within  the  source  selection  process  as  being  any  more  at  risk  than  any  other 
area,  it  is  difficult  to  ascertain  whether  there  is  a  specific  relationship  between  the  report 
results  and  the  observations  from  the  DPAP  database  of  peer-review  results.  The  obvious 
spike  in  Recommendations  in  Phase  1  can  be  generalized  as  an  issue  with  the  overall 
source-selection  process,  which  helps  to  corroborate  the  survey  results  on  which  the 
CCM  was  based. 

Two  areas  from  the  DPAP  database  of  peer-review  results  that  can  be  more 
directly  related  to  the  CCM  are  the  Pricing  and  Incentive  &  Award  Fee  categories.  Recall 
that  of  the  three  Types  of  Feedback,  Recommendations  comprised  89.13%  of  the 
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feedback  for  the  Pricing  category  and  86.96%  of  the  feedback  for  the  Incentive  &  Award 
Fee  category.  These  two  categories  garnered  the  highest  densities  of  Recommendations 
within  the  DPAP  database  of  peer-review  results,  providing  an  indication  that  these  areas 
are  presenting  difficulties  to  the  contracting  workforce.  This  also  shows  a  relationship  to 
the  Competency  Gaps  of  cost  and  price  analysis.  Unfortunately,  without  having  dates 
associated  with  the  DPAP  database  of  peer-review  entries,  it  is  difficult  to  ascertain 
whether  the  training  initiatives  that  were  driven  by  the  Department  of  Defense 
Contracting  Workforce  Competency  Assessment  Final  Report  (DPAP,  2007)  are  having  a 
noticeable  impact  on  the  number  of  Recommendations  that  these  two  categories  are 
attracting  from  the  peer-review  teams. 

The  final  area  identified  as  a  Competency  Gap  that  has  a  cross  reference  with  the 
DPAP  database  of  peer-review  results  is  the  number  of  Recommendations  within  the 
Post-award  1  Review  Phase,  albeit  tentatively  so.  Contract  performance  management  was 
identified  as  Competency  Gap,  which  would  align  with  the  Post-award  1  Review  Phase 
of  the  DPAP  process.  The  number  of  Recommendations  showed  an  uptick  between  Phase 
4,  with  no  recommendations,  to  11  Recommendations  in  the  Post-award  1  Phase.  The 
link  remains  tenuous,  however,  and  more  contracts  will  need  to  be  reviewed  in  the  Post¬ 
award  1  Phase  to  strengthen  the  tie  between  the  peer-review  results  and  the  Competency 
Gap.  The  following  section  addresses  themes  within  the  narrative  feedback  data  element 
of  the  DPAP  database  of  peer-review  results. 

D,  NARRATIVE  FEEDBACK  THEMES 

The  narrative  feedback  data  element  of  the  DPAP  database  of  peer-review  results 
provides  an  opportunity  for  the  peer-review  team  to  enter  its  comments  on  the  occurrence 
documented.  Although  difficult  to  summarize  or  quantify  the  narratives,  some  themes  did 
occur  within  the  comments. 

The  most  startling,  if  not  alarming,  issue  is  the  episodes  that  reveal  direct 
violations  of  the  FAR  or  DFARS.  Concerns  existed  about  improperly  proposed  award 
fees,  award  fees  that  were  not  necessary,  and  offerors  being  allowed  to  propose  their  own 
fee  pool  percentage.  In  another  instance,  a  contracting  officer  attempted  to  provide  a 
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performance  incentive  for  a  project  in  progress  review  that  was  already  mandated  by  a 
DFARS  clause.  Proposals  with  vague  requirements  were  also  a  recurring  theme. 

Questions  over  training  adequacies  persisted  as  well.  Some  comments  revolved 
around  concerns  over  the  use  of  contracting  officer  representatives  and  whether  they  had 
received  proper  training  to  execute  their  intended  function.  One  training  comment  stood 
out  above  the  others;  “Minimal  source  selection  experience  within  the  government 
evaluation  team  has  resulted  in  heavy  reliance  on  contractor  expertise.  Recommend 
obtaining  assistance  from  the  DAU  in  developing  the  source  selection  framework” 
(DPAP,  2013a). 

The  narrative  comments  read  like  a  summary  of  GAO  and  Inspector  General  (IG) 
reports  that  have  detailed  the  deficiencies  with  contracting  over  the  last  several  years.  The 
disturbing  aspect  is  that  these  entries  represent  contracts  valued  at  $1  billion  and  above. 
Presumably,  with  such  a  high  dollar  value,  DOD  agencies  and  departments  would  have 
experienced  contracting  officers  and  dedicated  contracting  teams  developing  these 
contracts.  Yet  mistakes  that  would  be  associated  with  an  inexperienced  contracting  team 
are  appearing  in  the  feedback.  The  next  section  contains  recommendations  for 
improvements  to  the  DPAP  database  of  peer-review  results. 

E,  RECOMMENDATIONS  FOR  IMPROVEMENTS  TO  THE  DPAP 

DATABASE  OF  PEER-REVIEW  RESULTS 

1.  Introduction 

This  section  addresses  recommendations  for  improvements  to  the  DPAP 
database  of  peer-review  results  based  on  the  data  analysis  conducted.  The  following 
recommendations  would  improve  the  functionality  and  usefulness  of  the  DPAP  database 
of  peer-review  results.  An  additional  category  should  be  added  to  the  DPAP  database  of 
peer-review  results  to  provide  what  basic  contract  type  is  being  utilized  in  the  entry.  The 
existing  Contract  Type  data  element  should  be  repurposed  as  a  Requirement  Type  data 
element.  The  Type  of  Feedback  data  element  needs  to  add  another  category  to  address 
regulatory  requirements.  The  Source  Selection  Category  should  be  further  subdivided  to 
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address  the  phases  that  make  up  the  souree  seleetion  proeess.  Lastly,  labelling 
eonventions  aeross  DPAP  produeed  produets  should  be  eonsistent. 

2.  Addition  of  a  Data  Element 

The  first  reeommendation  is  to  add  another  data  element  to  the  DPAP  database  of 
peer-review  results.  Currently,  there  is  a  Contraet  Type  data  element,  but  it  does  not 
simply  list  the  eommonly  used  types  of  eost  and  fixed-type  eontraets.  The  eurrent 
Contraet  Type  field  should  be  relabeled  as  Contraet  Requirement  Type  to  retain 
deseriptive  value  and  eontext  for  the  entry  being  examined.  The  Contraet  Type  field 
should  be  repurposed,  eontaining  one  of  the  basie  eontraet  types.  As  the  DPAP  database 
of  peer-review  results  stands,  the  state  of  the  Contraet  Type  field  negatively  impaets  data 
analysis.  This  is  due  to  the  diffieulty,  if  not  down-right  impossibility,  of  identifying 
whieh  basie  eontraet  type  is  being  employed  in  regard  to  the  data  entry.  If  basie  eontraet 
types  were  listed,  then  basie  analyties  eould  be  eondueted  to  identify  whieh,  if  any, 
eontraet  types  are  garnering  more  negative  feedbaek  than  others. 

3.  Regulatory  Requirements 

The  seeond  reeommendation  addresses  the  Reeommendation  feedbaek  type.  The 
Reeommendations  make  up  the  majority  of  the  DPAP  database  of  peer-review  results 
entries  and  address  a  myriad  of  issues,  some  small,  some  large.  The  issue  that  needs  to  be 
addressed  is  Reeommendations  that  are  regulatory  in  nature.  For  example,  eonsider  this 
entry  “An  Award  Fee  plan  needs  to  eonform  with  the  FAR  Rule  16.401(e)  (DPAP, 
2013a).”  This  Reeommendation  targets  a  violation  of  a  regulation,  in  this  ease  FAR 
16.401(e);  thus,  this  partieular  Reeommendation  earries  far  more  weight  than  might 
normally  be  assoeiated  with  a  reeommendation.  To  eall  attention  to  regulatory  violations, 
the  DPAP  database  of  peer-review  results  needs  to  eontain  a  “regulatory”  Feedbaek  Type 
that  ean  be  used  to  pinpoint  sueh  violations.  It  would  provide  another  method  for 
polieymakers  to  determine  over  time  whieh  areas  of  the  FAR  are  habitually 
misunderstood  or  underutilized  so  training  ean  be  tailored  to  address  those  shorteomings. 
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4,  Subdivide  the  Source  Selection  Category 

The  Source  Selection  Category  would  benefit  from  additional  refinement,  adding 
to  the  overall  usefulness  of  the  DPAP  database  of  peer-review  results.  The  Source 
Selection  Category  contained  115  entries  out  of  the  288  total  entries  in  the  database. 
By  comparison,  Pricing  came  in  a  distant  second  at  46  entries.  Source  Selection  as  its 
being  used  currently  within  the  DPAP  database  of  peer-review  results  covers  four  distinct 
phases  of  the  contracting  process.  Those  phases  are  procurement  planning,  solicitation 
planning,  solicitation,  and  source  selection.  The  Source  Selection  category  should  be 
further  subdivided  into  categories  that  mirror  the  aforementioned  phases,  again  for  the 
purposes  of  providing  an  accurate  assessment  of  where  problems  are  occurring  within  the 
contracting  process  and  enabling  policy  makers  to  address  deficiencies. 

5,  Ensure  Labeling  Conventions  Are  Consistent 

The  final  recommendation  addresses  labeling  conventions.  Phases,  titles,  contract 
types,  and  so  forth  within  the  DPAP  database  of  peer-review  results  do  not  consistently 
match  up  well  with  other  contracting  related  products,  some  of  which  were  created  by  the 
DPAP  organization  itself.  For  example,  it  is  difficult  to  cross-reference  any  area  of  the 
DPAP  peer-review  database  with  the  CCM.  The  CCM  was  developed  by  DPAP  shortly 
before  the  peer-review  requirement  was  established  in  2008.  Assuming  the  DOD  is 
serious  about  improving  the  performance  of  its  contracting  workforce,  then  should  not 
labels  on  the  DPAP  database  of  peer-review  results  allow  a  user  to  cross-reference 
competencies  that  have  been  identified  as  critical  for  accomplishing  contracting 
functions?  The  subsequent  section  encompasses  the  summary  of  the  data  analysis 
chapter. 

F.  SUMMARY 

This  chapter  discusses  the  results  of  the  analysis  of  the  DPAP  peer  reviews. 

Trends  within  the  DPAP  database  of  peer-review  results  were  examined.  Frequencies  of 

occurrences  by  Category,  Type  of  Contract,  Review  Phase  and  Type  of  Feedback  were 

inspected.  The  Recommendation  Type  of  Feedback  received  additional  attention.  The 

data  showed  the  majority  of  feedback  within  the  DPAP  database  of  peer-review  results 
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occurred  in  the  Reeommendation  Feedbaek  Type,  and  in  the  initial  Review  Phases. 
Further  examination  reveals  that  the  majority  of  the  oecurrenees  of  the  Prieing,  Ineentive 
and  Award  Fee,  &  Requirements/PWS/SOW  Categories  eonsist  of  Reeommendation 
Feedbaek  Type.  Applieability  of  relationships  within  the  data  to  previously  identified 
Competeney  Gaps  within  the  DOD  eontraeting  workforee  was  examined.  Themes  were 
identified  within  the  narrative  feedbaek  eomments  that  showed  issues  requiring  eorreetive 
aetion  within  high-dollar  DOD  eontraets.  The  next  ehapter  summarizes  and  eoneludes  the 
researeh  and  provides  areas  for  further  research. 
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V.  SUMMARY,  CONCLUSIONS,  AND  AREAS 
FOR  FURTHER  RESEARCH 


A,  SUMMARY 

Since  the  early  1990s,  the  number  of  personnel  within  the  aequisition  workforee 
has  steadily  deelined.  Sinee  the  onset  of  the  overseas  eontingency  operations  in  2002, 
DOD  eontraeting  aetions  have  risen  astronomieally.  Additionally,  sinee  the  relianee  on 
eontraets  has  inereased,  DOD  shows  no  signs  of  deereasing  that  relianee  for 
aceomplishing  its  mission.  With  the  reduetion  in  the  workforee  and  the  rapid  inerease  in 
the  number  of  eontraet  aetions  and  the  amount  of  dollars  expended,  issues  within  the 
eontraeting  workforee  have  also  risen  dramatieally,  mostly  as  a  result  of  a  workforee  that 
is  either  too  inexperieneed  or  spread  too  thin  for  its  assigned  task.  DPAP  implemented 
the  independent  management  review  (or  peer  review)  requirement  in  2008  as  a  means  of 
attempting  to  eompensate  for  the  lack  of  experienced  eontraeting  personnel  within  its 
ranks.  The  next  seetion  examines  the  eonelusions  of  this  thesis. 

B,  CONCLUSIONS 

Research  Question  1:  Are  there  trends  within  the  peer-review  results  of 
DOD-level  peer-reviewed  contracts? 

Trends  were  identified  in  the  DPAP  database  of  peer-review  results.  Not 
surprisingly,  most  of  the  entries  eonsisted  of  Reeommendation  Type  Feedback,  at  65%  of 
the  entries;  however.  Best  Praetiee  Feedback  Type  was  represented  at  24%  and  Lessons 
Learned  at  11%.  This  demonstrates  the  DPAP  database  of  peer-review  results  potential  as 
a  tool  to  spread  effeetive  eontraeting  praetiees  throughout  DOD.  The  analysis  also 
showed  that  the  entries  were  largely  Souree  Seleetion  foeused,  110  out  of  288  entries 
pertained  to  Souree  Seleetion.  The  next  most  prevalent  oeeurrenee  within  Category  Type 
was  Prieing  with  46  entries.  Potential  issues  were  also  indieated  with  the  following 
Categories:  Terms  &  Conditions,  Requirements/PWS/SOW,  and  Ineentive  &  Award  Fee. 
Those  eategories  amassed  70%  or  greater  Reeommendation  Type  Feedback. 
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Research  Question  2:  Are  any  trends  identified  related  to  the  competency 
gaps  identified  in  the  Department  of  Defense  Contracting  Workforce  Competency 
Assessment  Final  Report? 

In  regard  to  trends  between  the  DPAP  database  of  peer-review  results  and 
competency  gaps  from  the  Department  of  Defense  Contracting  Workforce  Competency 
Assessment  Final  Report  (DPAP,  2007),  the  strongest  link  was  between  issues  with  Cost 
and  Pricing  Analysis  identified  in  the  Final  Report  and  the  DPAP  database  of  peer- 
review  results  Pricing  and  Incentive  &  Award  Fee  Categories.  Pricing  consisted  of  89% 
Recommendation  Feedback  Type,  and  Incentive  &  Award  Fee  consisted  of  87%. 

This  research  identified  trends  within  the  results  provided  by  the  peer-review 
teams  and  recorded  within  the  DPAP  database  of  peer-review  results.  The  results  of  the 
study  can  be  used  for  further  research  into  areas  such  as  identifying  recurring  problem 
areas,  identifying  training  gaps,  and  answering  other  “why”  type  questions  associated 
with  the  results  of  this  study.  This  study  was  by  no  means  intended  to  be  exhaustive  in 
nature,  but  rather  to  investigate  an  area  that  had  previously  received  little,  if  any, 
attention. 

The  DPAP  database  of  peer-review  results  is  a  good  tool  for  the  DOD  contracting 
workforce;  it  does  require  some  refinement  to  reach  its  full  potential,  as  well  as  the 
implementation  of  a  dedicated  feedback  mechanism  to  disseminate  the  peer-review 
results  back  to  the  contracting  workforce.  An  important  consideration  that  comes  into 
focus  is  the  fact  that  the  trends  show  the  same  issues  facing  lower  dollar  threshold 
contracting  actions  are  impacting  high-dollar  contracting  actions  as  well.  However,  since 
the  DPAP  database  of  peer-review  results  threshold  level  is  above  $1  billion,  the  effects 
of  mistakes  in  contracting  are  magnified. 

C.  AREAS  FOR  FURTHER  RESEARCH 

As  DPAP’s  database  of  peer-review  results  accumulates  entries  over  time,  the 
statistical  population  for  analysis  will  increase.  The  current  number  of  entries  within  the 
DPAP  database  of  peer-review  results  allows  for  trend  and  frequency  analysis  within  the 
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database  itself.  However,  onee  the  various  fields  are  broken  down  into  their  eonstituent 
parts,  more  entries  would  allow  for  more  meaningful  comparisons. 

Specifically,  there  are  two  areas  to  consider.  First,  as  Best  Practice  feedback  types 
continue  to  accumulate,  is  there  a  relationship  with  the  Contract  Management  Maturity 
Model  (CMMM)?  CMMM  assessments  contain  assessments  of  process  strength,  process 
results,  and  management  support.  Do  the  narrative  comments  in  the  DPAP  database  of 
peer-review  results  best  practice  entries  support  any  of  those?  Additionally,  if  service- 
level  peer-review  results  could  be  obtained,  a  more  direct  comparison  could  be  made  with 
respective  CMMM  assessments  from  that  service.  Second,  since  the  DPAP  database  of 
peer-review  results  is  a  relatively  new  construct,  as  more  contracts  enter  the  Post  Award 
1  review  phase,  will  we  continue  to  see  an  uptick  in  recommendations  (see  Figure  11)? 
Could  this  indicate  a  continuation  of  poor  contractor  surveillance  practices  previously 
noted  by  the  GAO? 

The  Source  Selection  category  received  the  most  attention  overall  by  the  peer- 
review  teams  with  115  out  of  288  entries.  The  source  selection  process  receives  a  good 
deal  of  emphasis  in  training  and  in  practice  by  the  contracting  community,  while  the 
contract  administration  phase  is  sometimes  considered  to  be  neglected.  An  interview- 
based  study  with  members  of  peer-review  teams  could  potentially  identify  whether  the 
Source  Selection  category  receives  so  much  attention  in  the  DPAP  database  of  peer- 
review  results  because  it  is  simply  better  understood  than  the  other  categories. 

The  peer-review  process  will  continue  to  evolve  at  the  DOD  level  and  the  service 
component  level.  As  the  process  evolves,  the  lessons  provided  by  the  peer-review  process 
should  be  disseminated  back  out  to  the  contracting  workforce.  As  this  occurs,  will 
sustained  protest  rates  decrease? 

The  service  components  are  mandated  by  the  peer-review  policy  to  execute  peer 
reviews  at  the  service  level  for  lower  dollar  threshold  contract  actions.  Are  the  services 
complying  with  this  requirement?  Are  they  compiling  a  database  similar  to  the  DPAP 
database  of  peer-review  results?  Can  the  peer-review  results  be  obtained  for  further 
analysis?  The  Army  FAR  supplement,  the  AFARS,  provides  more  detail  on  the  peer- 
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review  requirement.  The  AFARS  assigns  responsibility  for  peer  reviews  of  solieitations 
and  eontraets  valued  at  greater  than  $50  million  to  a  Solieitation  Review  Board  (SRB) 
and  Contraet  Review  Board  (CRB).  The  Prineipal  Assistant  Responsible  for  Contraeting 
(PARC)  has  responsibility  for  establishing  peer  -review  proeedures  for  eontraets  valued 
at  less  than  $50  million. 

The  AFARS  5 101. 170(1  )(e)  also  mandates  that: 

The  SRB/CRB  will  be  an  independent,  multi-funetional  team  eomprised  of 
senior  level  experts,  whieh  will  at  a  minimum  inelude  representatives  from 
the  aequisition  eenter,  small  business  offiee,  offiee  of  eounsel, 
requirements  eommunity,  and  in  the  ease  of  non-eompetitive  aetions,  the 
eompetition  advoeate.  (AFARS,  2014) 

From  that  definition,  it  would  seem  evident  that  the  Army  is  attempting  to  garner 
insights  from  all  angles  of  the  aequisition  proeess  into  the  peer-review  proeess.  The 
AFARS  also  stipulates  that  the  same  members  will  take  part  in  both  the  SRB  and  CRB; 
although  a  good  idea,  proeurement  lead-times  might  often  prevent  this  from  happening 
given  typieal  military  permanent  ehange  of  station  (PCS)  eyeles.  There  is  also  striet 
language  on  who  will  ehair  the  boards,  and  there  is  no  ability  to  delegate  the  authority. 
The  PARC  is  required  to  ehair  SRB/CRB  for  aetions  valued  between  $50  million  and 
$250  million,  while  the  Head  of  the  Contraeting  Aetivity  (HCA)  has  responsibility  for  the 
$250  million  to  $1  billion  range.  Interestingly,  per  AFARS  5101.170(l)(e),  PARCs  and 
the  HCA  may  waive  the  requirement  for  a  formal  review  board.  The  waiver  is  required  to 
be  in  writing  and  ineluded  in  the  eontraet  file.  Data  on  the  number  of  peer  reviews 
eondueted,  along  with  any  issues  or  trends,  best  praetiees,  or  relevant  feedbaek  on  the 
proeess  itself,  is  required  to  be  submitted  quarterly  (along  with  information  on  waivers  of 
the  review  proeess)  to  the  Proeurement  Poliey  and  Support  Direetorate  (AFARS,  2014). 

AFARS  5101.170(2)  eontinues  with  further  direetion  on  the  handling  of  post¬ 
award  peer  reviews.  Post-award  peer  reviews  for  eontraets  valued  at  between  $500 
million  and  $1  billion  will  be  eondueted  for  serviees  eontraets  that  were  approved  by  a 
Headquarters,  Department  of  the  Army  (HQDA)-level  Army  Serviees  Strategy  Panel 
(ASSP).  The  post-award  review  minimum  review  requirements  are  as  follows: 

•  Contraet  performanee  in  terms  of  eost,  sehedule,  and  requirements. 
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•  Use  of  contracting  mechanisms,  including  the  use  of  competition,  the 
contract  structure  and  type  the  definition  of  contract  requirements,  cost  or 
pricing  methods,  the  award  and  negotiation  of  task  orders,  and 
management  and  oversight  mechanisms. 

•  Contractor’s  use,  management,  and  oversight  of  subcontractors. 

•  Staffing  of  contract  management  and  oversight  functions 

•  Extent  of  any  pass-through  and  excessive  pass-through  charges  by  the 
contractor  (as  defined  in  Section  852  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2007,  Public  Law  109-364). 

ASSP  panel  members  are  to  address  whether  or  not  the  services  review  structure 
is  effective,  evaluate  the  current  acquisition  as  to  adequacy  of  competition,  provide  an 
assessment  of  contract  performance,  assess  the  conduct  of  the  government  surveillance 
plan,  and  pass  on  any  lessons  learned  or  best  practices  for  use  on  other  ongoing 
acquisitions  (AFARS,  2014).  Additionally,  the  Army  Contracting  Command  Desk  Book 
(May  2012  edition),  the  handbook  for  Army  contracting  officers,  provides  the 
requirements  for  peer  reviews  listed  from  the  AFARS.  The  only  additional  guidance  it 
provides  are  two  toolkits  for  pre-award  and  post-award  peer  reviews.  The  toolkits  list 
documents  required  by  the  OSD-level  peer-review  team  and  elements  required  to  be 
confirmed  by  the  OSD  peer-review  team  (ACC,  2012). 

Finally,  interviews  could  be  conducted  with  DPAP  personnel  to  obtain  time- 
phased  data  about  DPAP  database  of  peer-review  results  entries.  The  data  could  then  be 
analyzed  to  determine  if  the  competency  gaps  identified  in  the  Department  of  Defense 
Contracting  Worlforce  Competency  Assessment  Final  Report  (DPAP,  2007)  are  closing. 
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APPENDIX.  FIGURES  AND  TABLES 
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Figure  1.  DOD  Acquisition  Trends  (from  Gansler,  2007) 
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Figure  2.  Defense  Acquisition  Workforce  Retirement  Eligibility  Distribution 

(from  DOD,  2010,  p.2-22) 
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Figure  3.  DOD  Contract  Obligations  and  Acquisition  Workforce  Trends 

(from  GAO,  2006,  p.4) 
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DoD  C'liteii;)  for  the  Argiiixitioii  of  SeiAires  (Pre-.w.u  (t) 


1 

Tenet 

Review  Crttaria: 

Primarily 

Assessed  at: 

1 

Acquisition  Sirstegy:  Service  acquisiticins  shouid  have  a  comprehensive  acquisition  strategy  that 
reflects  program  objectives,  leverages  spend  data  to  amve  at  strategic  sourcing  soiutions  for  the 
enteqmse  being  supported,  incorporates  strategic  contractng  tools,  is  deveioped  poor  to  the 
issuance  of  a  soiiotation  (amended  as  appicable)  and  is  adiered  to  throughout  performance 

Detailed  written  and 
approved  acquisiton 
strategy 

Review/Approval  of 
Acquisition  Strategy 

2 

Clearty  Defined  Requirements:  Service  acquisitions  should  use  a  performance  work  statemerrt  or 

statement  of  obiedives  that  clearly  defines  the  services  the  program  seeks  to  receive 

Requirements  clearty 
staled 

Peer  Review 

3 

Period  of  Performance:  Contract  penod  of  performance  should  reflect  a  length  that  is  consistent 

with  technologcal  dependence,  industry  standards,  and  sufiaent  tme  to  reclaim  program  ownership 
(in  cases  with  an  acquisition  history  of  a  single  provider)  such  that  fair  competition  can  occur 

Typically  this  is  3-5  years  with  certain  exceptions  (e  g  performance-based  logsbcs  and  energy- 
savings  performance  contracts)  Sin^e-award  contracts  for  knowledge  based  services  should  be 
linvled  to  three  years  (includng  options)  unless,  by  exceptxm.  fully  justified  fa  lon«r  penods 

Length  of  contract 
appropnate 

Review/Approval  of 
Acquisition  Strategy 

4 

Appropriate  Contract  Type:  Services  acquisitions  should  be  predsposedt^varoCF^F  or  CPiF 

arrangements  when  robust  competition  or  recent  competitve  pnong  history  does  not  exist  to  build 
suffiaent  cost  knc^edge  of  those  services  within  that  market  segment  V^hen  robust  competition 
exists,  athere  IS  recent  competitive  pnong  history.  FFPshouldtypicallybeused  Award  fee 
contracts  should  be  linited  as  they  provide  United  motivabon  for  cost  dsoplne  T&M  is  the  least 
preferable  contracttype  and  must  be  fustrfiedwhen  used  and  limited  (e  q  <  10%  of  contact  value) 

Contract  type 
appropnate 

Review/Approval  of 
Acguisibon  Strategy 

5 

Sodo-Cconomic  Considerations:  The  Department  hi^ly  values  small  business  coninbutions  and 

expects  maximum  opportunities  fa  small  business  participation 

Small  business 

opporiunibes 

Review/Approval  of 

Acguisibon  Strategy 

6 

Participation  Decision  Points:  Acquisitions  with  longer  penods  of  performance  paibcularty 

multiple  award  contracts,  should  indude  dedsion  ponts  (on  and  offramps)  to  ensure  a  qualified  pool 
of  contradas  that  will  provide  contiiuous  service  throur^out  the  life  of  the  contrad 

Decision  pants  (on  and 
off-ramps)  fa  longer 
lemi  contracts 

Revtew/Approval  of 
Acguisibon  Strate(^ 

7 

Competition:  Service  acquisition  requirements  shouid  be  articulated  in  such  a  way  to  provide  for 

maximum  competibon  and.  for  muHfile  award  contracts,  throu^outthe  life  ofthe  contract  with 
meaningful  competition  for  orders  Evaluabon  factors  are  tied  to  key  program  reguirements 

Approach  provides  for 
robust  competibon 

Review/ Approval  of 
Acguisibon  StratecN 

8 

Objective  Incentives:  Objedive  cntena  will  be  utilized,  whenever  pospble.  to  measure  contract 

perfonnance  Service  contracts  valued  at  $1  bilion  or  mae  shall  indude  provisions  to  adiieve 
producbvitv  imaovements  and  cost  efficiendes  throudiout  the  contract  penod 

Award/Incentive  Fee 

set  up  to  reward 
effedive  outcomes 

Review/Approval  of 
Acguisibon  Strategy 

9 

Inherently  Governmental  Functions  Acquisitions  ^a  services  must  addessthe  extent  of  the 

agency's  reliance  on  contractas  to  perform  fundions  closely  associated  with  inherently 
governmental  functions  as  required  by  FAR  Subpart  7  5.  and  DFARS  Subpart  207  5 

Required 
determinations 
appropriately  executed 

Peer  Review 

10 

Conflict  of  Interests:  lAhen  one  contractawill  provide  oversigjit  fa  anaher  contrada  a  dired 

woilc  to  another  contracta.  the  acquisition  documentaton  should  address  measures  to 
reduce/eliminate  paential  conflid  of  interest 

Evaluate  finanoal 

interest  of  support 
contractors,  as  requred 

Peer  Review 

11 

Perfonnance  flilanegemenb  Service  acquisrtions  should  utikze  performance  based  charadenstics 

to  the  maximum  extent  practcabie  to  include  measures  that  are  diredly  tied  to  program  objedwes 
These  measures  should  include  consideration  of  program  cost 

Meaningful 

performance  measures 

Peer  Review 

12 

Contract  Surveillanc»:  Service  acquisitions  must  have  assigied  contracting  officer  representabves 

(CORs)  that  use  tailored  quality  assurance  surveillance  plans  to  monita  contracta  performance 

Appropnate  program 
oversight 

Peer  Review 

Figure  4.  DOD  Peer-Review  Criteria  for  Services  Acquisitions  during  the 
Pre-award  Phase  (from  DPAP,  2013c) 


60 


DoD  f 'liteiin  for  the  Arqiiisitioii  of  SeiAires  (Postnwni  d) 


Tenet 

Review  Criteria: 

Primarily 

Assessed  at: 

1 

Contractor  Performance  Assessment:  Service  acquisitions  undergo  penodc  reviews  to  ensure 
the  program  is  on  course  with  respect  to  cost  schedule  and  performance  requirements,  and  any 
necessary  adjustments  are  made 

Program  meets  or 
exceeds  establi^ed 
cost,  schedule  and 
performance  entena 

Peer  Review 

2 

Maintaining  Competition:  Multfite  award  service  ac(|jislions  use  contractng  mechanisms, 
including  the  use  of  competition,  the  contract  structure  and  type,  the  definition  of  contract 
requirements,  cost  or  pricng  methods,  the  award  and  negotation  of  task  orders  ,  and  management 
and  oversi^t  mechanisms  In  competitve  procurements  (to  include  multiple  award  IDIQ  competition 
for  task  orders),  offerors  must  be  given  a  reasonable  amount  of  time  to  submt  a  proposal  (typically 

30  days  or  more).  If  suffiaent  time  is  provided  for  offerors  to  respond  and  yet  only  one  offer  is 
received  in  a  compettive  procurement  confractno  officers  should  not  depend  on  the  standard  at 

FAR  15  403-1(cXii)  m  determining  the  pnce  to  be  ^r  and  reasonable 

Robust  competition  for 
orders,  appropnate 
contract  type,  well- 
defined  requirements, 
appropnate  cost/pndng 
methods 

Peer  Review 

3 

Contractor’s  Subcontract  Management:  Service  acquistions  undergo  penocic  reviews  to  monitor 
the  contractor's  use.  management,  and  oversir^t  of  subcontractors 

Contractors 
subcontract 
management  evaluated 

Peer  Review 

4 

Contract  Surveillance:  Service  acquisitions  undergo  penodc  reviews  to  ensure  the  appropnate 
staffing  of  Government  contract  management  andoversKtit  functions  to  include  CORs 

Appropnate 

Government  oversi^t 
personnel  in  place 

Peer  Review 

5 

Assessment  of  Excessive  Pass-Through  Charges:  Service  acquisrtions  undergo  penodic  review 

to  evaluate  the  extent  of  any  pass-throu^s.  and  excessive  pass-throu^  charges  (as  defined  in 
DFARS  252  215-7003/4).  by  the  contractor 

Contractor's  pass¬ 

through  charges 
evaluated 

Peer  Review 

6 

Inherently  Governmental  Functions:  For  service  acquislions  under  which  one  contractor  provides 

oversight  for  services  performed  by  other  contractors,  penodic  reviews  evaluate  the  extent  of  the 
agency's  reliance  on  the  contractato  perform  acquisrbon  functions  closely  associated  with  inherently 
governmental  functions  as  defined  in  section  2383(b)(3)  of  title  10.  United  States  Code 

Evaluate  appropnate 
performance  of 
contractor  functions  (as 
required) 

Peer  Review 

7 

Conflicts  of  Interest  (Financial):  For  service  acquisitions  under  which  one  contracta  provides 

oversight  for  services  performed  by  other  contractors,  penodic  reviews  evaluate  the  financial  interest 
of  any  pnme  contractor  performng  acquisition  functions  descnbed  in  paragraph  six  in  any  contract  or 
subcontract  with  regard  to  which  the  contractor  provided  advice  or  recommendations  to  the  agency 

Evaluate  finanaal 
interest  of  support 
contractors  (as 
required) 

Peer  Review 

8 

Objective  Incentives:  For  contracts  with  award  and/w  incentive  fee.  objectrve  cntenawill  be 
utilized,  whenever  possible,  to  measure  contract  performance  VSliere  elective  entena  exist  and  it 

IS  appropnate  to  also  incentivize  subjective  elements  of  performance,  the  most  appropnate  con^ct 
type  would  be  a  multiple  incentive  type  contract  containing  both  incentive  and  award  fee  critena 

Award/incentive  fees 
are  paid  in  a  manner 
consistent  with  DoO 
policy,  or  with  a 
corrective  aefion  plan  n 
place  to  address  past 
inconsistencies 

Peer  Review 

Figure  5.  DOD  Peer-Review  Criteria  for  Serviees  Aequisitions  during  the 
Post-award  Phase  (from  DPAP,  2013c) 
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OSO  level  Peer  fteviewt 

Best  Prectices,  Insom  Uamtd.  and  R«co<nmen<Ut»om  (m  of  December  20. 2013) 


CaMfocy 

Type  of  Contract 

feadbatfc 

Review  Phase 

Type  of  Feedback 

muntweand 

Award  Fee 

Logistiul  SprviCK 
(Competitiw) 

Sofiotabon  contamed  provisnn  for  an  award-term.  Peer  Review  Team  (PRT)  recommended  addme 
languaie  to  indcate  itut  an  award  of  icpilem*  dost  not  pjarantw  pterosc  of  an  award  term  option  by 
the  Government 

Phase  1 

Recommendation 

IncentMand 

Award  Fee 

Competitive  ServKet 
Comract 

The  planned  contract  structure  provides  tor  a  Cost-Plus-tncentive^ee  (CPIF)  arranpement  with  iiKentives 
only  rciatirvf  to  cost.  The  Peer  Review  Team  (PRT)  recommends  eKentrAbnf  schedule  and/or  technical 
performance  as  wel  as  cost  The  nefotiated  incentive  arranfement  could  take  eito  aaount  the 
contractor's  projected  cash  flow. 

Phase  2 

Recommendation 

Incentn^and 

Award  Fee 

Non-Competitive  Services 
Contract 

Compared  to  other  types  of  contractors.  Federaly  Funded  Research  ft  Development  Centers  (FFRDCs) 
have  unique  fee  arran{ements.  wheh  are  discussicd  in  Df  ARS  21S.404.7S.  Fee  Requirements,  typicafiy 
referred  to  as  'lee-for-need.*  DFARS  states  that  FFRDCs  may  be  provided  no  fee  and  that  the  decision  to 
pay  fee  rest  solely  with  the  contracting  offer. 

Phase  1 

Recommendation 

Incentive  artd 

Award  Fee 

Competitive  Muft^ik  Award 
(Combmabon  of  firm-fixed 
price  and  cost  type  fine 
Items) 

Award  Fee  Plan  (AFP)  does  not  reflect  the  best  business  arrangement  tor  the  government  in  the  tolowing 
areas:  a)  Rod-Over  of  award  lee  amounts;  b)  Provisional  fees.  Therefore,  the  acquisition  team  was 
encouraged  to  consider  developwg  cost  and  performance  inontives  in  ficu  of  award  fees  in  accordance 
with  DoDpoficy. 

Phase  1 

Recommendation 

Incent^and 

Award  Fee 

Competitive  Muft^le  Award 
Services  Contract 

A>  draftad.  oNcrerx  wil  prepew  a  fc«  peel  percentagr  and  th«  rvaluation  tcam/PreeurAg  Contracting 
Officer  (PCO)  must  determtoe  whether  or  not  the  offer  is  ‘balanced.’  The  Peer  Review  Team  (PRT) 
suggested  defining  the  fee  pool  percentage  and  not  allowing  the  offerors  to  propose  a  percentage.  This 
ensures  there  is  an  adequate  pool  and  simpfifies  the  evaluation  of  proposals  by  efimiruting  the  need  to 
evaluate  'balana.' 

Phase  1 

Recommendabon 

Incentive  and 

Award  Fee 

Non-Competitrve  Weapon 
System 

The  Peer  Review  Team  (PRT)  recommended  the  contracting  officer  ensure  that  cost  is  not  considered  in 
both  incentive  fee  and  award  fee.  that  maximum  use  of  obyectiM  criteria  be  used  to  develop  award  fee 
entena  whenever  posuble.  and  that  pertormarKe  incentives  are  dupkeated  in  the  award  fee  criteria. 
Although  the  draft  award  fee  plan  incoiporates  some  obfective  measure,  there  is  stf  much  vagueness  in 
the  wordng  and  more  measures  of  processes  than  outcomes. 

Phase  2 

Recommendation 

Incentive  artd 

Award  Fee 

Non-Competitive  Weapon 
System 

The  amount  or  percent  alocated  tor  each  performance  iiKentive  event  should  be  gradual  in  nature,  so  the 
fees  are  not  front 4oadcd  and  ample  fee  is  left  through  contract  completion.  In  addrtKjn,  dear  definition  of 
success  tor  all  events  must  be  communicated  to  the  contractor.  For  example,  the  definition  for  exit  and 
entrance  criteria  and  what  comprises  an  M  (In  Process  Review)  was  not  noted  in  the  attachment  referred 
to  in  the  solicitation.  Ukewne.  we  recommend  the  Contracting  Officer  consider  whether  performtog  an  IBR 
with  180  days,  whichisrequiredby  DFARS  clause  2S2.234-7002(i).  should  be  part  of  any  performance 
incentive.  If  it  d.  the  resulting  fee  percent  allocated  to  K  should  be  minimal  and  the  aitena  for  successful 
completion  dearfy  spelled  out  in  the  resulting  comract  (as  currently,  the  information  relating  to  the  IBR  is 
scant). 

Phase2 

Recommendabon 

Figure  6.  Excerpt  From  DPAP  Database  for  OSD-Level  Peer  Reviews 

(from  DPAP,  2013  a) 


DP.\P  Peer 

Re\iew  Phases 

1)  Prior  to  issuance  cf  the  solicitation 

(Xo  comparable 
step) 

2)  Prior  to  request  for  final  proposal 
resisions  &  3)  Prior  to  contract  award 

4)  Post-award  resiew 

Contracting 

Process  Phases 

1)  Procurement 
Plaimii^ 

2]  Solicitation 
Planniig 

3)  SoHciahon 

4)  Source  Selection 

5)  Contract 
.Administration 

6)  Contract 
Ooseout 

Figure  7.  DPAP  Peer-Review  Phases  Compared  to  the  Contracting  Process  Phases 
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Terms  &  Conditions 

Source  Selection 
Requirements/PWS/SOW 
Pricing 
Post  Award  Admin 

Peer  Review 

Market  Research 

Incentive  and  Award  Fee 


Occurrences  by  Category  Type 


0  20  40  60  80  100  120 


■  Occurrences 


Figure  8.  Frequency  Distribution  of  Category  Type  (after  DPAP,  2013a) 
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Occurrences  by  Contract  Type 


Weapon  System  Technology  Demonstration 
Weapon  System  Production  Lot  Buy  (Sole  Source) 
Weapon  System  Development  Sole  Source 
Weapon  System  Development 
Sole  Source  Procurement  of  Services 
Non  Competitive  Weapon  System  Development 
Non  Competitive  Weapon  System 
Non  Competitive  Supplies  Contract 
Non  Competitive  Services  Contract 
Non  Competitive  Multiple  Award  Services  Contract 
Multiple  Award  Contract  for  R&D 
Multiple  Award  Construction  Contract 
Logistical  Services  (Competitive) 
Design/Build  Construction  Contract 
Competitive  Weapons  System  Development 
Competitive  Supplies  Contract 
Competitive  Services  Contract 
Competitive  Procurement  of  Supplies 
Competitive  Procurement  of  Services 
Competitive  Multiple  Award  IDIQ  Supplies  Contract 
Competitive  Multiple  Award  IDIQ 
Competitive  Multiple  Award  Services  Contract 
Competitive  Multiple  Award  IDIQ  Services 
Competitive  Multiple  Award  IDIQ  R&D  Contract 
Competitive  Multiple  Award  (Combination  Multiple- 

Competitive  IQC 
Competitive  IDIQ  Services  Contract 
Competitive  IDIQ  for  IT  Products 
Competitive  IDIQ 
Competitive  Design/Build  FPIF 
Commodity  (Competitive) 
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0  10  20  30  40  50  60  70 


■  Occurrences 


Figure  9.  Frequency  Distribution  of  Contract  Type  (after  DPAP,  2013a) 
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Occurrences  by  Review  Phase 


I  Occurrences 


Figure  10.  Frequency  Distribution  of  Review  Phase  (after  DPAP,  2013a) 


Occurrences  by  Feedback  Type 


Lessons  Learned 


Best  Practice 


Recommendation 


■  Occurrences 


Figure  1 1 .  Frequency  Distribution  of  Type  of  Feedback  (after  DPAP,  2013a) 
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Recommendation  Occurrences  By  Phase 


Figure  12.  Frequency  Distribution  of  Recommendations  by  Phase 

(after  DPAP,  2013a) 


Occurrences  of  Recommendation 
Within  Categories 


Terms  &  Conditions 
Source  Selection 
Requirements/PWS/SOW 
Pricing 
Post  Award  Admin 
Incentive  and  Award  Fee 
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0  10  20  30  40  50  60  70 


■  Occurrences  of  Recommendation 


Figure  13.  Frequency  Distribution  of  Recommendations  within  Categories 

(after  DPAP,  2013a) 
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Occurrences  of  Recommendation  Type  Feedback  by 
Contract  Type 


Weapon  System  Technology  Demonstration 
Weapon  System  Production  Lot  Buy  (Sole  Source) 
Weapon  System  Development  Sole  Source 
Weapon  System  Development 
Sole  Source  Procurement  of  Services 
Non  Competitive  Weapon  System  Development 
Non  Competitive  Weapon  System 
Non  Competitive  Supplies  Contract 
Non  Competitive  Services  Contract 
Non  Competitive  Multiple  Award  Services  Contract 
Multiple  Award  Contract  for  R&D 
Multiple  Award  Construction  Contract 
Logistical  Services  (Competitive) 
Design/Build  Construction  Contract 
Competitive  Weapons  System  Development 
Competitive  Supplies  Contract 
Competitive  Services  Contract 
Competitive  Procurement  of  Supplies 
Competitive  Procurement  of  Services 
Competitive  Multiple  Award  IDIQ  Supplies  Contract 
Competitive  Multiple  Award  IDIQ 
Competitive  Multiple  Award  Services  Contract 
Competitive  Multiple  Award  IDIQ  Services 
Competitive  Multiple  Award  IDIQ  R&D  Contract 
Competitive  Multiple  Award  (Combination  Multiple  Award... 

Competitive  IQC 
Competitive  IDIQ  Services  Contract 
Competitive  IDIQ  for  IT  Products 
Competitive  IDIQ 
Competitive  Design/Build  FPIF 
Commodity  (Competitive) 


■  Occurrences  of  Recommendation 


Figure  14.  Frequency  Distribution  of  Recommendations  by  Contract  Type 

(after  DPAP,  2013a) 
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Table  1.  Percentage  of  Recommendations  by  Category  (after  DPAP,  2013a) 


Categories 

Total 

Occurrences 

Occurrences  of 
Recommendation 

Percent  of 
Recommendation 

Incentive  and  Award  Fee 

23 

20 

86.95652174  % 

Market  Research 

4 

0 

0% 

Peer  Review 

1 

0 

0% 

Post-award  Admin 

35 

9 

25.71428571  % 

Pricing 

46 

41 

89.13043478  % 

Requirements/P  W  S/ 
SOW 

24 

17 

70.83333333  % 

Source  Selection 

115 

66 

57.39130435  % 

Terms  &  Conditions 

40 

37 

92.5  % 

Total 

288 

190 

65.97222222  % 
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Table  2.  Percentage  of  Recommendations  by  Contract  Type  (after  DPAP,  2013a) 


Contract  Type 

Total 

Occurrences  of 
Recommendation 

Percentage  of 
Recommendation 

Commodity  (Competitive) 

4 

3 

75 

Competitive  Design/Build  FPIF 

2 

2 

100 

Competitive  IDIQ 

1 

1 

100 

Competitive  IDIQ  for  IT  Products 

4 

4 

100 

Competitive  IDIQ  Services  Contract 

6 

2 

33.33333333 

Competitive  IQC 

1 

1 

100 

Competitive  Multiple  Award  (Combination 
Multiple  Award  FFP  and  Cost  type  line  items) 

7 

7 

100 

Competitive  Multiple  Award  IDIQ  R&D  Contract 

1 

0 

0 

Competitive  Multiple  Award  IDIQ  Services 

22 

3 

13.63636364 

Competitive  Multiple  Award  Services  Contract 

59 

41 

69.49152542 

Competitive  Multiple  Award  IDIQ 

5 

2 

40 

Competitive  Multiple  Award  IDIQ  Supplies 

2 

1 

50 

Contract 

Competitive  Procurement  of  Services 

20 

9 

45 

Competitive  Procurement  of  Supplies 

14 

9 

64.28571429 

Competitive  Services  Contract 

39 

16 

41.02564103 

Competitive  Supplies  Contract 

1 

1 

100 

Competitive  Weapons  System  Development 

16 

14 

87.5 

Design/Build  Construction  Contract 

3 

3 

100 

Logistical  Services  (Competitive) 

10 

7 

70 

Multiple  Award  Construction  Contract 

2 

2 

100 

Multiple  Award  Contract  for  R&D 

6 

5 

83.33333333 

Non-Competitive  Multiple  Award  Services 

11 

11 

100 

Contract 

Non-Competitive  Services  Contract 

4 

4 

100 

Non-Competitive  Supplies  Contract 

10 

8 

80 

Non-Competitive  Weapon  System 

8 

8 

100 

Non-Competitive  Weapon  System  Development 

6 

5 

83.33333333 

Sole-Source  Procurement  of  Services 

1 

1 

100 

Weapon  System  Development 

5 

4 

80 

Weapon  System  Development  Sole  Source 

1 

0 

0 

Weapon  System  Production  Lot  Buy  (Sole  Source) 

3 

3 

100 

Weapon  System  Technology  Demonstration 

14 

13 

92.85714286 

Total 

288 

190 

65.97222222 
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Table  3. 


DOD  Contracting  Competency  Model  (after  DPAP,  2007) 


D<}D  Ccxilr«Kling  Comp*‘lency  Mcxit*l 
cttflbNthcd  March  ?007 

Avwara  arc 

Ccsemraaoo  Hon*  Bast  T o  Satn-V 

Consida’a  SocO-^conooK  RegU^ntans 

Pronrc»  Compancr 

Sotrca  S«iocKfl  Punnng 

SdctaDon  ofOfc’s 

Resf craMi^  Deemnaocr 

Bd  Evetjatcr  (SaaieC  Bocng 

Proposal  Evaluaton  iConradnp  by  NeqaCsSon) 

Source  Selectofl 

CoofactAwa’C 

Process  Proess 

C«v«top  src/or  Nc^oeae  Fosocrs 

j-Giica»ncfC<n®f  T3r  FUarc  "per  cenpexon 

Terrs  and  Condoons 

rTf^.-waicr  arid  tJ^gr-rcr 

Advanc«  Cca  ancyo'  Pnce  Anaiysa 

ACvarceC  Cost  aod’or  Pace  Aralyso 

Confact  AtrvDtraOon 

InSatcc  at  iworl 

Contact  Feffcrmance  Varaqerv^' 

Issue  craroes  arid  Modicaicrs 

r'Cprcve  PayrenPccuesB 

cioMOiKConrax: 

Sffiai  Bisuwss/soooeaxKXTic  Prograin: 

Aoereaang  Saioi  Butoess  Corcems 

riogofaia  Forwa'C  Prong  Raa  AQrgQrT«re 
arc  Adnrr>osf  r  Cost  Accoutw;  Swcards 

NegotaM  PerMrO  Pidrig  Rasa  Agreeaierts 
and  AijTwiser  Cost  Acer  urrrvi  Swidards 

Corcaa  Tcrwtaior 

Confact  Terrwasofl 

Proaire(tieftFo:c/ 

Proorenenf  Ana  yjtj 

Ctfvtr  Conip««roM 

E-Susress  and  AuianaieC  TcOb 

Acb/tr  Program  Coordnaior  br  Purc*ia»e  CaiC 

Const’udors’Afcmact  and  Ercineeinq 

Ccrcracinj  ir>  a  Corcnj^T;  aid'or  CtxitatErvt’orcwri: 

ConrasCng  n  a  CcrVigart  artcltor  Con*ai  Envtronaiort 

=ro<ess<)nel  Ccrnpeiency 

Prooicn  s«r/ng 

Cujjjffier  Service 

Oral  Conmumcaaon 

Wreer  ConMojrKaaon 

InKrperserd  SSAs 

Cecist/enass 

Technical  CrecKiay 

PMlOMy 

Resaerce 

Accounacaty 
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